2026-02-26T03:00:23.462 INFO:root:teuthology version: 1.2.4.dev11+g6e3d0419c 2026-02-26T03:00:23.480 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-26T03:00:23.566 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71147 branch: main description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm} perf_ignorelist settings/optimized workloads/fio_4K_rand_rw} email: null first_in_suite: false flavor: debug job_id: '71147' kernel: branch: distro kdb: true sha1: distro last_in_suite: false machine_type: trial name: shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: crimson cpu num: 4 crimson osd obc lru size: 10 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore omap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true seastore cachepin size pershard: 64M seastore main device type: RANDOM_BLOCK_SSD seastore max concurrent transactions: 8 crimson_compat: true flavor: debug log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH sha1: abffb617683951511fccf3c3cab6e219f2ec83a8 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: debug sha1: abffb617683951511fccf3c3cab6e219f2ec83a8 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996 owner: scheduled_shraddhaag@soko04.front.sepia.ceph.com priority: 101 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 5117 sha1: abffb617683951511fccf3c3cab6e219f2ec83a8 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: 1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996 targets: trial053.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFp7yIyMFWu15PV8HBCA04CtZcJvdNOSgFjUcqQKEETY0bb8EkATuuC3ZGu4eapyulIJoMRRXAyt6P3JpRVzBlw= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - install: null - ceph: conf: osd: debug monc: 20 wait-for-scrub: false - ssh_keys: null - cbt: benchmarks: librbdfio: iodepth: - 32 log_avg_msec: 100 mode: - randrw norandommap: true op_size: - 4096 osd_ra: - 4096 pool_profile: rbd procs_per_volume: - 1 time: 60 vol_size: 4096 volumes_per_client: - 2 cluster: iterations: 1 osds_per_node: 3 pool_profiles: rbd: pg_size: 128 pgp_size: 128 replication: 3 user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6e3d0419ccca0e75a03e08feb708946e198236f4 timestamp: 2026-02-26_02:22:56 tube: trial user: shraddhaag verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699 2026-02-26T03:00:23.566 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996/qa; will attempt to use it 2026-02-26T03:00:23.567 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996/qa/tasks 2026-02-26T03:00:23.567 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-26T03:00:23.568 INFO:teuthology.task.internal:Checking packages... 2026-02-26T03:00:23.569 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'debug' and ceph hash 'abffb617683951511fccf3c3cab6e219f2ec83a8' 2026-02-26T03:00:23.569 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-26T03:00:23.569 INFO:teuthology.packaging:ref: None 2026-02-26T03:00:23.569 INFO:teuthology.packaging:tag: None 2026-02-26T03:00:23.569 INFO:teuthology.packaging:branch: main 2026-02-26T03:00:23.569 INFO:teuthology.packaging:sha1: abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:00:23.569 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&ref=main 2026-02-26T03:00:23.697 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5593.g63656871 2026-02-26T03:00:23.699 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-26T03:00:23.714 INFO:teuthology.task.internal:no buildpackages task found 2026-02-26T03:00:23.714 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-26T03:00:23.728 INFO:teuthology.task.internal:Saving configuration 2026-02-26T03:00:23.744 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-26T03:00:23.758 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-26T03:00:23.839 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial053.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71147', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-26 02:56:55.465650', 'locked_by': 'scheduled_shraddhaag@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFp7yIyMFWu15PV8HBCA04CtZcJvdNOSgFjUcqQKEETY0bb8EkATuuC3ZGu4eapyulIJoMRRXAyt6P3JpRVzBlw='} 2026-02-26T03:00:23.839 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-26T03:00:23.851 INFO:teuthology.task.internal:roles: ubuntu@trial053.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-26T03:00:23.851 INFO:teuthology.run_tasks:Running task console_log... 2026-02-26T03:00:23.926 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbc9e7b1da0>, signals=[15]) 2026-02-26T03:00:23.926 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-26T03:00:23.937 INFO:teuthology.task.internal:Opening connections... 2026-02-26T03:00:23.937 DEBUG:teuthology.task.internal:connecting to ubuntu@trial053.front.sepia.ceph.com 2026-02-26T03:00:23.938 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T03:00:24.001 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-26T03:00:24.012 DEBUG:teuthology.orchestra.run.trial053:> uname -m 2026-02-26T03:00:24.074 INFO:teuthology.orchestra.run.trial053.stdout:x86_64 2026-02-26T03:00:24.074 DEBUG:teuthology.orchestra.run.trial053:> cat /etc/os-release 2026-02-26T03:00:24.129 INFO:teuthology.orchestra.run.trial053.stdout:NAME="CentOS Stream" 2026-02-26T03:00:24.129 INFO:teuthology.orchestra.run.trial053.stdout:VERSION="9" 2026-02-26T03:00:24.129 INFO:teuthology.orchestra.run.trial053.stdout:ID="centos" 2026-02-26T03:00:24.129 INFO:teuthology.orchestra.run.trial053.stdout:ID_LIKE="rhel fedora" 2026-02-26T03:00:24.129 INFO:teuthology.orchestra.run.trial053.stdout:VERSION_ID="9" 2026-02-26T03:00:24.129 INFO:teuthology.orchestra.run.trial053.stdout:PLATFORM_ID="platform:el9" 2026-02-26T03:00:24.129 INFO:teuthology.orchestra.run.trial053.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-26T03:00:24.130 INFO:teuthology.orchestra.run.trial053.stdout:ANSI_COLOR="0;31" 2026-02-26T03:00:24.130 INFO:teuthology.orchestra.run.trial053.stdout:LOGO="fedora-logo-icon" 2026-02-26T03:00:24.130 INFO:teuthology.orchestra.run.trial053.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-26T03:00:24.130 INFO:teuthology.orchestra.run.trial053.stdout:HOME_URL="https://centos.org/" 2026-02-26T03:00:24.130 INFO:teuthology.orchestra.run.trial053.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-26T03:00:24.130 INFO:teuthology.orchestra.run.trial053.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-26T03:00:24.130 INFO:teuthology.orchestra.run.trial053.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-26T03:00:24.131 INFO:teuthology.lock.ops:Updating trial053.front.sepia.ceph.com on lock server 2026-02-26T03:00:24.214 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-26T03:00:24.227 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-26T03:00:24.239 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-26T03:00:24.240 DEBUG:teuthology.orchestra.run.trial053:> test '!' -e /home/ubuntu/cephtest 2026-02-26T03:00:24.254 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-26T03:00:24.266 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-26T03:00:24.266 DEBUG:teuthology.orchestra.run.trial053:> test -z $(ls -A /var/lib/ceph) 2026-02-26T03:00:24.311 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-26T03:00:24.408 INFO:teuthology.run_tasks:Running task kernel... 2026-02-26T03:00:24.428 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-26T03:00:24.428 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-26T03:00:24.429 DEBUG:teuthology.orchestra.run.trial053:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-26T03:00:24.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:00:24.443 DEBUG:teuthology.orchestra.run.trial053:> uname -r 2026-02-26T03:00:24.499 INFO:teuthology.orchestra.run.trial053.stdout:5.14.0-681.el9.x86_64 2026-02-26T03:00:24.499 INFO:teuthology.task.kernel:Running kernel on trial053: 5.14.0-681.el9.x86_64 2026-02-26T03:00:24.499 DEBUG:teuthology.orchestra.run.trial053:> sudo yum install -y kernel 2026-02-26T03:00:25.212 INFO:teuthology.orchestra.run.trial053.stdout:CentOS Stream 9 - BaseOS 27 MB/s | 8.9 MB 00:00 2026-02-26T03:00:27.834 INFO:teuthology.orchestra.run.trial053.stdout:CentOS Stream 9 - AppStream 13 MB/s | 27 MB 00:02 2026-02-26T03:00:30.538 INFO:teuthology.orchestra.run.trial053.stdout:CentOS Stream 9 - CRB 29 MB/s | 8.0 MB 00:00 2026-02-26T03:00:31.432 INFO:teuthology.orchestra.run.trial053.stdout:CentOS Stream 9 - Extras packages 137 kB/s | 20 kB 00:00 2026-02-26T03:00:33.186 INFO:teuthology.orchestra.run.trial053.stdout:Extra Packages for Enterprise Linux 12 MB/s | 20 MB 00:01 2026-02-26T03:00:36.437 INFO:teuthology.orchestra.run.trial053.stdout:lab-extras 1.4 MB/s | 50 kB 00:00 2026-02-26T03:00:37.462 INFO:teuthology.orchestra.run.trial053.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-26T03:00:37.462 INFO:teuthology.orchestra.run.trial053.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-26T03:00:37.482 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:00:37.484 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:00:37.484 INFO:teuthology.orchestra.run.trial053.stdout: Package Architecture Version Repository Size 2026-02-26T03:00:37.484 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:00:37.484 INFO:teuthology.orchestra.run.trial053.stdout:Installing: 2026-02-26T03:00:37.484 INFO:teuthology.orchestra.run.trial053.stdout: kernel x86_64 5.14.0-686.el9 baseos 901 k 2026-02-26T03:00:37.484 INFO:teuthology.orchestra.run.trial053.stdout:Installing dependencies: 2026-02-26T03:00:37.484 INFO:teuthology.orchestra.run.trial053.stdout: kernel-core x86_64 5.14.0-686.el9 baseos 17 M 2026-02-26T03:00:37.484 INFO:teuthology.orchestra.run.trial053.stdout: kernel-modules x86_64 5.14.0-686.el9 baseos 40 M 2026-02-26T03:00:37.484 INFO:teuthology.orchestra.run.trial053.stdout: kernel-modules-core x86_64 5.14.0-686.el9 baseos 31 M 2026-02-26T03:00:37.484 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:00:37.484 INFO:teuthology.orchestra.run.trial053.stdout:Transaction Summary 2026-02-26T03:00:37.484 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:00:37.485 INFO:teuthology.orchestra.run.trial053.stdout:Install 4 Packages 2026-02-26T03:00:37.485 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:00:37.485 INFO:teuthology.orchestra.run.trial053.stdout:Total download size: 89 M 2026-02-26T03:00:37.485 INFO:teuthology.orchestra.run.trial053.stdout:Installed size: 135 M 2026-02-26T03:00:37.485 INFO:teuthology.orchestra.run.trial053.stdout:Downloading Packages: 2026-02-26T03:00:39.041 INFO:teuthology.orchestra.run.trial053.stdout:(1/4): kernel-5.14.0-686.el9.x86_64.rpm 603 kB/s | 901 kB 00:01 2026-02-26T03:00:57.783 INFO:teuthology.orchestra.run.trial053.stdout:(2/4): kernel-core-5.14.0-686.el9.x86_64.rpm 870 kB/s | 17 MB 00:20 2026-02-26T03:01:14.306 INFO:teuthology.orchestra.run.trial053.stdout:(3/4): kernel-modules-core-5.14.0-686.el9.x86_6 900 kB/s | 31 MB 00:35 2026-02-26T03:01:23.293 INFO:teuthology.orchestra.run.trial053.stdout:(4/4): kernel-modules-5.14.0-686.el9.x86_64.rpm 892 kB/s | 40 MB 00:45 2026-02-26T03:01:23.295 INFO:teuthology.orchestra.run.trial053.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:01:23.295 INFO:teuthology.orchestra.run.trial053.stdout:Total 1.9 MB/s | 89 MB 00:45 2026-02-26T03:01:23.489 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction check 2026-02-26T03:01:23.689 INFO:teuthology.orchestra.run.trial053.stdout:Transaction check succeeded. 2026-02-26T03:01:23.689 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction test 2026-02-26T03:01:24.072 INFO:teuthology.orchestra.run.trial053.stdout:Transaction test succeeded. 2026-02-26T03:01:24.072 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction 2026-02-26T03:01:25.183 INFO:teuthology.orchestra.run.trial053.stdout: Preparing : 1/1 2026-02-26T03:01:25.490 INFO:teuthology.orchestra.run.trial053.stdout: Installing : kernel-modules-core-5.14.0-686.el9.x86_64 1/4 2026-02-26T03:01:25.512 INFO:teuthology.orchestra.run.trial053.stdout: Installing : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-26T03:01:28.041 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-26T03:01:28.073 INFO:teuthology.orchestra.run.trial053.stdout: Installing : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-26T03:01:31.113 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-26T03:01:31.118 INFO:teuthology.orchestra.run.trial053.stdout: Installing : kernel-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:01:34.161 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:01:51.481 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:01:51.501 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:01:51.575 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: kernel-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:01:51.575 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : kernel-5.14.0-686.el9.x86_64 1/4 2026-02-26T03:01:51.575 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-26T03:01:51.575 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-26T03:01:51.632 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:01:51.632 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:01:51.632 INFO:teuthology.orchestra.run.trial053.stdout:Installed: 2026-02-26T03:01:51.632 INFO:teuthology.orchestra.run.trial053.stdout: kernel-5.14.0-686.el9.x86_64 2026-02-26T03:01:51.632 INFO:teuthology.orchestra.run.trial053.stdout: kernel-core-5.14.0-686.el9.x86_64 2026-02-26T03:01:51.632 INFO:teuthology.orchestra.run.trial053.stdout: kernel-modules-5.14.0-686.el9.x86_64 2026-02-26T03:01:51.632 INFO:teuthology.orchestra.run.trial053.stdout: kernel-modules-core-5.14.0-686.el9.x86_64 2026-02-26T03:01:51.632 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:01:51.632 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:01:51.693 DEBUG:teuthology.orchestra.run.trial053:> rpm -q kernel | sort -rV | head -n 1 2026-02-26T03:01:51.722 INFO:teuthology.orchestra.run.trial053.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-26T03:01:51.722 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-26T03:01:51.722 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-681.el9.x86_64 Expected: 5.14.0-686.el9.x86_64 2026-02-26T03:01:51.722 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-26T03:01:51.722 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-26T03:01:51.722 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-26T03:01:51.722 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial053.front.sepia.ceph.com, path=None, version=distro) 2026-02-26T03:01:51.722 DEBUG:teuthology.orchestra.run.trial053:> rpm -q kernel | sort -rV | head -n 1 2026-02-26T03:01:51.786 INFO:teuthology.orchestra.run.trial053.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-26T03:01:51.787 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-26T03:01:51.787 DEBUG:teuthology.orchestra.run.trial053:> sudo rpm -qi grub2-tools 2026-02-26T03:01:51.855 INFO:teuthology.orchestra.run.trial053.stdout:Name : grub2-tools 2026-02-26T03:01:51.855 INFO:teuthology.orchestra.run.trial053.stdout:Epoch : 1 2026-02-26T03:01:51.855 INFO:teuthology.orchestra.run.trial053.stdout:Version : 2.06 2026-02-26T03:01:51.856 INFO:teuthology.orchestra.run.trial053.stdout:Release : 120.el9 2026-02-26T03:01:51.856 INFO:teuthology.orchestra.run.trial053.stdout:Architecture: x86_64 2026-02-26T03:01:51.856 INFO:teuthology.orchestra.run.trial053.stdout:Install Date: Wed 18 Feb 2026 05:18:42 PM UTC 2026-02-26T03:01:51.856 INFO:teuthology.orchestra.run.trial053.stdout:Group : Unspecified 2026-02-26T03:01:51.856 INFO:teuthology.orchestra.run.trial053.stdout:Size : 9108611 2026-02-26T03:01:51.856 INFO:teuthology.orchestra.run.trial053.stdout:License : GPLv3+ 2026-02-26T03:01:51.856 INFO:teuthology.orchestra.run.trial053.stdout:Signature : RSA/SHA256, Thu 04 Dec 2025 09:42:59 AM UTC, Key ID 05b555b38483c65d 2026-02-26T03:01:51.856 INFO:teuthology.orchestra.run.trial053.stdout:Source RPM : grub2-2.06-120.el9.src.rpm 2026-02-26T03:01:51.856 INFO:teuthology.orchestra.run.trial053.stdout:Build Date : Wed 03 Dec 2025 12:10:38 PM UTC 2026-02-26T03:01:51.856 INFO:teuthology.orchestra.run.trial053.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-26T03:01:51.856 INFO:teuthology.orchestra.run.trial053.stdout:Packager : builder@centos.org 2026-02-26T03:01:51.856 INFO:teuthology.orchestra.run.trial053.stdout:Vendor : CentOS 2026-02-26T03:01:51.857 INFO:teuthology.orchestra.run.trial053.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-26T03:01:51.857 INFO:teuthology.orchestra.run.trial053.stdout:Summary : Support tools for GRUB. 2026-02-26T03:01:51.857 INFO:teuthology.orchestra.run.trial053.stdout:Description : 2026-02-26T03:01:51.857 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:01:51.857 INFO:teuthology.orchestra.run.trial053.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-26T03:01:51.857 INFO:teuthology.orchestra.run.trial053.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-26T03:01:51.857 INFO:teuthology.orchestra.run.trial053.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-26T03:01:51.857 INFO:teuthology.orchestra.run.trial053.stdout:hardware devices. 2026-02-26T03:01:51.857 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:01:51.857 INFO:teuthology.orchestra.run.trial053.stdout:This subpackage provides tools for support of all platforms. 2026-02-26T03:01:51.858 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-26T03:01:51.859 INFO:teuthology.task.kernel:Updating grub on trial053 to boot 5.14.0-686.el9.x86_64 2026-02-26T03:01:51.859 DEBUG:teuthology.orchestra.run.trial053:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-26T03:01:51.923 INFO:teuthology.orchestra.run.trial053.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-26T03:01:51.925 DEBUG:teuthology.orchestra.run.trial053:> sudo command -v grubby && echo yes || echo no 2026-02-26T03:01:51.989 INFO:teuthology.orchestra.run.trial053.stdout:/sbin/grubby 2026-02-26T03:01:51.990 INFO:teuthology.orchestra.run.trial053.stdout:yes 2026-02-26T03:01:51.990 DEBUG:teuthology.orchestra.run.trial053:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-686.el9.x86_64" | head -n 1 2026-02-26T03:01:52.054 INFO:teuthology.orchestra.run.trial053.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64.conf 2026-02-26T03:01:52.055 DEBUG:teuthology.orchestra.run.trial053:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64 2026-02-26T03:01:52.126 DEBUG:teuthology.orchestra.run.trial053:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-26T03:01:52.336 INFO:teuthology.orchestra.run.trial053.stderr:Generating grub configuration file ... 2026-02-26T03:01:52.401 INFO:teuthology.orchestra.run.trial053.stderr:Found linux image: /boot/vmlinuz-5.14.0-686.el9.x86_64 2026-02-26T03:01:52.408 INFO:teuthology.orchestra.run.trial053.stderr:Found initrd image: /boot/initramfs-5.14.0-686.el9.x86_64.img 2026-02-26T03:01:52.655 INFO:teuthology.orchestra.run.trial053.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-26T03:01:52.661 INFO:teuthology.orchestra.run.trial053.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-26T03:01:52.695 INFO:teuthology.orchestra.run.trial053.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-26T03:01:52.702 INFO:teuthology.orchestra.run.trial053.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-26T03:01:52.733 INFO:teuthology.orchestra.run.trial053.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-26T03:01:52.739 INFO:teuthology.orchestra.run.trial053.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-26T03:01:52.762 INFO:teuthology.orchestra.run.trial053.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-26T03:01:52.769 INFO:teuthology.orchestra.run.trial053.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-26T03:01:52.855 INFO:teuthology.orchestra.run.trial053.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-26T03:01:52.863 INFO:teuthology.orchestra.run.trial053.stderr:done 2026-02-26T03:01:52.865 DEBUG:teuthology.orchestra.run.trial053:> test -d /sys/firmware/efi 2026-02-26T03:01:52.881 DEBUG:teuthology.orchestra.run.trial053:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-26T03:01:52.946 INFO:teuthology.orchestra.run.trial053.stdout:/boot/efi/EFI/BOOT 2026-02-26T03:01:52.946 INFO:teuthology.orchestra.run.trial053.stdout:/boot/efi/EFI/centos 2026-02-26T03:01:52.948 DEBUG:teuthology.orchestra.run.trial053:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-26T03:01:53.015 INFO:teuthology.orchestra.run.trial053.stdout:no 2026-02-26T03:01:53.016 DEBUG:teuthology.orchestra.run.trial053:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-26T03:01:53.079 INFO:teuthology.orchestra.run.trial053.stdout:yes 2026-02-26T03:01:53.079 DEBUG:teuthology.orchestra.run.trial053:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-26T03:01:53.144 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial053 2026-02-26T03:01:53.144 DEBUG:teuthology.orchestra.run.trial053:> sudo bash -c ' 2026-02-26T03:01:53.145 DEBUG:teuthology.orchestra.run.trial053:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-26T03:01:53.145 DEBUG:teuthology.orchestra.run.trial053:> ' 2026-02-26T03:02:23.187 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-26T03:02:23.188 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial053.front.sepia.ceph.com' 2026-02-26T03:02:23.188 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T03:03:23.249 DEBUG:teuthology.orchestra.remote:timed out 2026-02-26T03:03:32.257 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial053.front.sepia.ceph.com' 2026-02-26T03:03:32.258 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T03:03:32.392 DEBUG:teuthology.orchestra.run.trial053:> true 2026-02-26T03:03:32.544 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial053.front.sepia.ceph.com' 2026-02-26T03:03:32.544 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-686.el9.x86_64"... 2026-02-26T03:03:32.545 DEBUG:teuthology.orchestra.run.trial053:> uname -r 2026-02-26T03:03:32.601 INFO:teuthology.orchestra.run.trial053.stdout:5.14.0-686.el9.x86_64 2026-02-26T03:03:32.601 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-686.el9.x86_64 vs 5.14.0-686.el9.x86_64 2026-02-26T03:03:32.601 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-26T03:03:32.601 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-26T03:03:33.602 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-26T03:03:33.602 DEBUG:teuthology.orchestra.run.trial053:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-26T03:03:33.632 INFO:teuthology.orchestra.run.trial053.stdout:ttyS1 2026-02-26T03:03:33.649 DEBUG:teuthology.parallel:result is None 2026-02-26T03:03:33.649 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-26T03:03:33.662 INFO:teuthology.task.internal:Creating test directory... 2026-02-26T03:03:33.662 DEBUG:teuthology.orchestra.run.trial053:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-26T03:03:33.703 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-26T03:03:33.714 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-26T03:03:33.728 INFO:teuthology.task.internal:Creating archive directory... 2026-02-26T03:03:33.728 DEBUG:teuthology.orchestra.run.trial053:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-26T03:03:33.766 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-26T03:03:33.777 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-26T03:03:33.777 DEBUG:teuthology.orchestra.run.trial053:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-26T03:03:33.824 INFO:teuthology.orchestra.run.trial053.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-26T03:03:33.833 INFO:teuthology.orchestra.run.trial053.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-26T03:03:33.835 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-26T03:03:33.846 INFO:teuthology.task.internal:Configuring sudo... 2026-02-26T03:03:33.847 DEBUG:teuthology.orchestra.run.trial053:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-26T03:03:33.899 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-26T03:03:33.920 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-26T03:03:33.920 DEBUG:teuthology.orchestra.run.trial053:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-26T03:03:33.956 DEBUG:teuthology.orchestra.run.trial053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-26T03:03:34.122 DEBUG:teuthology.orchestra.run.trial053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-26T03:03:34.147 DEBUG:teuthology.orchestra.run.trial053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-26T03:03:34.285 DEBUG:teuthology.orchestra.run.trial053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-26T03:03:34.309 DEBUG:teuthology.orchestra.run.trial053:> set -ex 2026-02-26T03:03:34.309 DEBUG:teuthology.orchestra.run.trial053:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-26T03:03:34.372 DEBUG:teuthology.orchestra.run.trial053:> sudo service rsyslog restart 2026-02-26T03:03:34.441 INFO:teuthology.orchestra.run.trial053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-26T03:03:34.867 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-26T03:03:34.880 INFO:teuthology.task.internal:Starting timer... 2026-02-26T03:03:34.880 INFO:teuthology.run_tasks:Running task pcp... 2026-02-26T03:03:34.899 INFO:teuthology.run_tasks:Running task selinux... 2026-02-26T03:03:34.915 DEBUG:teuthology.orchestra.run.trial053:> sudo service auditd rotate 2026-02-26T03:03:34.943 INFO:teuthology.orchestra.run.trial053.stdout:Rotating logs: 2026-02-26T03:03:34.945 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-26T03:03:34.945 DEBUG:teuthology.orchestra.run.trial053:> /usr/sbin/getenforce 2026-02-26T03:03:35.001 INFO:teuthology.orchestra.run.trial053.stdout:Permissive 2026-02-26T03:03:35.001 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial053.front.sepia.ceph.com': 'permissive'} 2026-02-26T03:03:35.001 DEBUG:teuthology.orchestra.run.trial053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-26T03:03:35.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:03:35.065 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-26T03:03:35.065 DEBUG:teuthology.orchestra.run.trial053:> sudo /usr/sbin/setenforce permissive 2026-02-26T03:03:35.129 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-26T03:03:35.142 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-26T03:03:35.149 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-26T03:03:35.150 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial053.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-26T03:06:15.095 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial053.front.sepia.ceph.com')] 2026-02-26T03:06:15.096 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial053.front.sepia.ceph.com' 2026-02-26T03:06:15.096 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T03:06:15.181 DEBUG:teuthology.orchestra.run.trial053:> true 2026-02-26T03:06:15.255 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial053.front.sepia.ceph.com' 2026-02-26T03:06:15.255 INFO:teuthology.run_tasks:Running task clock... 2026-02-26T03:06:15.271 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-26T03:06:15.271 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-26T03:06:15.271 DEBUG:teuthology.orchestra.run.trial053:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-26T03:06:15.322 INFO:teuthology.orchestra.run.trial053.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-26T03:06:15.333 INFO:teuthology.orchestra.run.trial053.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-26T03:06:15.354 INFO:teuthology.orchestra.run.trial053.stderr:sudo: ntpd: command not found 2026-02-26T03:06:15.364 INFO:teuthology.orchestra.run.trial053.stdout:506 Cannot talk to daemon 2026-02-26T03:06:15.375 INFO:teuthology.orchestra.run.trial053.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-26T03:06:15.387 INFO:teuthology.orchestra.run.trial053.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-26T03:06:15.436 INFO:teuthology.orchestra.run.trial053.stderr:bash: line 1: ntpq: command not found 2026-02-26T03:06:15.456 INFO:teuthology.orchestra.run.trial053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-26T03:06:15.456 INFO:teuthology.orchestra.run.trial053.stdout:=============================================================================== 2026-02-26T03:06:15.456 INFO:teuthology.orchestra.run.trial053.stdout:^? ntp.shastacoe.net 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-26T03:06:15.457 INFO:teuthology.orchestra.run.trial053.stdout:^? vps-lga1.orleans.ddnss.de 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-26T03:06:15.457 INFO:teuthology.orchestra.run.trial053.stdout:^? 23.186.168.130 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-26T03:06:15.457 INFO:teuthology.orchestra.run.trial053.stdout:^? tick.srs1.ntfo.org 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-26T03:06:15.457 INFO:teuthology.run_tasks:Running task install... 2026-02-26T03:06:15.468 DEBUG:teuthology.task.install:project ceph 2026-02-26T03:06:15.468 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}} 2026-02-26T03:06:15.468 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'} 2026-02-26T03:06:15.468 INFO:teuthology.task.install:Using flavor: debug 2026-02-26T03:06:15.471 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-26T03:06:15.471 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-26T03:06:15.471 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8', 'tag': None, 'wait_for_package': False} 2026-02-26T03:06:15.472 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:06:15.612 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/ 2026-02-26T03:06:15.612 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5312.gabffb617 2026-02-26T03:06:15.935 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-26T03:06:15.935 DEBUG:teuthology.orchestra.run.trial053:> set -ex 2026-02-26T03:06:15.935 DEBUG:teuthology.orchestra.run.trial053:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-26T03:06:15.961 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-26T03:06:15.961 DEBUG:teuthology.orchestra.run.trial053:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-26T03:06:16.026 DEBUG:teuthology.orchestra.run.trial053:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-26T03:06:16.099 DEBUG:teuthology.orchestra.run.trial053:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-26T03:06:16.161 INFO:teuthology.orchestra.run.trial053.stdout:check_obsoletes = 1 2026-02-26T03:06:16.162 DEBUG:teuthology.orchestra.run.trial053:> sudo yum clean all 2026-02-26T03:06:16.310 INFO:teuthology.orchestra.run.trial053.stdout:41 files removed 2026-02-26T03:06:16.323 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install ceph-radosgw 2026-02-26T03:06:16.826 INFO:teuthology.orchestra.run.trial053.stdout:ceph packages for x86_64 265 kB/s | 94 kB 00:00 2026-02-26T03:06:17.113 INFO:teuthology.orchestra.run.trial053.stdout:ceph noarch packages 94 kB/s | 24 kB 00:00 2026-02-26T03:06:17.242 INFO:teuthology.orchestra.run.trial053.stdout:ceph source packages 20 kB/s | 2.1 kB 00:00 2026-02-26T03:06:18.050 INFO:teuthology.orchestra.run.trial053.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.9 MB 00:00 2026-02-26T03:06:21.646 INFO:teuthology.orchestra.run.trial053.stdout:CentOS Stream 9 - AppStream 8.8 MB/s | 27 MB 00:03 2026-02-26T03:06:33.095 INFO:teuthology.orchestra.run.trial053.stdout:CentOS Stream 9 - CRB 899 kB/s | 8.0 MB 00:09 2026-02-26T03:06:34.899 INFO:teuthology.orchestra.run.trial053.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 20 kB 00:01 2026-02-26T03:06:35.274 INFO:teuthology.orchestra.run.trial053.stdout:Extra Packages for Enterprise Linux 65 MB/s | 20 MB 00:00 2026-02-26T03:06:38.432 INFO:teuthology.orchestra.run.trial053.stdout:lab-extras 2.5 MB/s | 50 kB 00:00 2026-02-26T03:06:39.687 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:06:39.688 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:06:39.689 INFO:teuthology.orchestra.run.trial053.stdout: Package Arch Version Repository Size 2026-02-26T03:06:39.689 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:06:39.689 INFO:teuthology.orchestra.run.trial053.stdout:Installing: 2026-02-26T03:06:39.689 INFO:teuthology.orchestra.run.trial053.stdout: ceph-radosgw x86_64 2:20.3.0-5312.gabffb617.el9 ceph 11 M 2026-02-26T03:06:39.689 INFO:teuthology.orchestra.run.trial053.stdout:Upgrading: 2026-02-26T03:06:39.689 INFO:teuthology.orchestra.run.trial053.stdout: librados2 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 3.3 M 2026-02-26T03:06:39.689 INFO:teuthology.orchestra.run.trial053.stdout: librbd1 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 2.7 M 2026-02-26T03:06:39.689 INFO:teuthology.orchestra.run.trial053.stdout:Installing dependencies: 2026-02-26T03:06:39.690 INFO:teuthology.orchestra.run.trial053.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-26T03:06:39.690 INFO:teuthology.orchestra.run.trial053.stdout: ceph-base x86_64 2:20.3.0-5312.gabffb617.el9 ceph 5.1 M 2026-02-26T03:06:39.690 INFO:teuthology.orchestra.run.trial053.stdout: ceph-common x86_64 2:20.3.0-5312.gabffb617.el9 ceph 21 M 2026-02-26T03:06:39.690 INFO:teuthology.orchestra.run.trial053.stdout: ceph-selinux x86_64 2:20.3.0-5312.gabffb617.el9 ceph 25 k 2026-02-26T03:06:39.690 INFO:teuthology.orchestra.run.trial053.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-26T03:06:39.690 INFO:teuthology.orchestra.run.trial053.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-26T03:06:39.690 INFO:teuthology.orchestra.run.trial053.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-26T03:06:39.690 INFO:teuthology.orchestra.run.trial053.stdout: libcephfs2 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 859 k 2026-02-26T03:06:39.690 INFO:teuthology.orchestra.run.trial053.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-26T03:06:39.690 INFO:teuthology.orchestra.run.trial053.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-26T03:06:39.690 INFO:teuthology.orchestra.run.trial053.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-26T03:06:39.691 INFO:teuthology.orchestra.run.trial053.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-26T03:06:39.691 INFO:teuthology.orchestra.run.trial053.stdout: libradosstriper1 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 263 k 2026-02-26T03:06:39.691 INFO:teuthology.orchestra.run.trial053.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-26T03:06:39.691 INFO:teuthology.orchestra.run.trial053.stdout: librgw2 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 5.6 M 2026-02-26T03:06:39.691 INFO:teuthology.orchestra.run.trial053.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-26T03:06:39.691 INFO:teuthology.orchestra.run.trial053.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-26T03:06:39.691 INFO:teuthology.orchestra.run.trial053.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-26T03:06:39.691 INFO:teuthology.orchestra.run.trial053.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-26T03:06:39.691 INFO:teuthology.orchestra.run.trial053.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-26T03:06:39.691 INFO:teuthology.orchestra.run.trial053.stdout: python3-ceph-argparse x86_64 2:20.3.0-5312.gabffb617.el9 ceph 45 k 2026-02-26T03:06:39.691 INFO:teuthology.orchestra.run.trial053.stdout: python3-ceph-common x86_64 2:20.3.0-5312.gabffb617.el9 ceph 185 k 2026-02-26T03:06:39.691 INFO:teuthology.orchestra.run.trial053.stdout: python3-cephfs x86_64 2:20.3.0-5312.gabffb617.el9 ceph 203 k 2026-02-26T03:06:39.692 INFO:teuthology.orchestra.run.trial053.stdout: python3-rados x86_64 2:20.3.0-5312.gabffb617.el9 ceph 324 k 2026-02-26T03:06:39.692 INFO:teuthology.orchestra.run.trial053.stdout: python3-rbd x86_64 2:20.3.0-5312.gabffb617.el9 ceph 305 k 2026-02-26T03:06:39.692 INFO:teuthology.orchestra.run.trial053.stdout: python3-rgw x86_64 2:20.3.0-5312.gabffb617.el9 ceph 100 k 2026-02-26T03:06:39.692 INFO:teuthology.orchestra.run.trial053.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-26T03:06:39.692 INFO:teuthology.orchestra.run.trial053.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-26T03:06:39.692 INFO:teuthology.orchestra.run.trial053.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-26T03:06:39.692 INFO:teuthology.orchestra.run.trial053.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-26T03:06:39.692 INFO:teuthology.orchestra.run.trial053.stdout:Installing weak dependencies: 2026-02-26T03:06:39.692 INFO:teuthology.orchestra.run.trial053.stdout: libcephfs-daemon x86_64 2:20.3.0-5312.gabffb617.el9 ceph 37 k 2026-02-26T03:06:39.692 INFO:teuthology.orchestra.run.trial053.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 26 k 2026-02-26T03:06:39.692 INFO:teuthology.orchestra.run.trial053.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-26T03:06:39.693 INFO:teuthology.orchestra.run.trial053.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-26T03:06:39.693 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:06:39.693 INFO:teuthology.orchestra.run.trial053.stdout:Transaction Summary 2026-02-26T03:06:39.693 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:06:39.693 INFO:teuthology.orchestra.run.trial053.stdout:Install 35 Packages 2026-02-26T03:06:39.693 INFO:teuthology.orchestra.run.trial053.stdout:Upgrade 2 Packages 2026-02-26T03:06:39.693 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:06:39.694 INFO:teuthology.orchestra.run.trial053.stdout:Total download size: 61 M 2026-02-26T03:06:39.694 INFO:teuthology.orchestra.run.trial053.stdout:Downloading Packages: 2026-02-26T03:06:41.609 INFO:teuthology.orchestra.run.trial053.stdout:(1/37): ceph-base-20.3.0-5312.gabffb617.el9.x86 3.0 MB/s | 5.1 MB 00:01 2026-02-26T03:06:41.621 INFO:teuthology.orchestra.run.trial053.stdout:(2/37): ceph-selinux-20.3.0-5312.gabffb617.el9. 2.0 MB/s | 25 kB 00:00 2026-02-26T03:06:41.635 INFO:teuthology.orchestra.run.trial053.stdout:(3/37): libcephfs-daemon-20.3.0-5312.gabffb617. 2.6 MB/s | 37 kB 00:00 2026-02-26T03:06:41.649 INFO:teuthology.orchestra.run.trial053.stdout:(4/37): libcephfs-proxy2-20.3.0-5312.gabffb617. 1.8 MB/s | 26 kB 00:00 2026-02-26T03:06:41.710 INFO:teuthology.orchestra.run.trial053.stdout:(5/37): libcephfs2-20.3.0-5312.gabffb617.el9.x8 14 MB/s | 859 kB 00:00 2026-02-26T03:06:41.732 INFO:teuthology.orchestra.run.trial053.stdout:(6/37): libradosstriper1-20.3.0-5312.gabffb617. 11 MB/s | 263 kB 00:00 2026-02-26T03:06:42.065 INFO:teuthology.orchestra.run.trial053.stdout:(7/37): ceph-common-20.3.0-5312.gabffb617.el9.x 9.6 MB/s | 21 MB 00:02 2026-02-26T03:06:42.079 INFO:teuthology.orchestra.run.trial053.stdout:(8/37): ceph-radosgw-20.3.0-5312.gabffb617.el9. 5.1 MB/s | 11 MB 00:02 2026-02-26T03:06:42.080 INFO:teuthology.orchestra.run.trial053.stdout:(9/37): python3-ceph-argparse-20.3.0-5312.gabff 3.0 MB/s | 45 kB 00:00 2026-02-26T03:06:42.097 INFO:teuthology.orchestra.run.trial053.stdout:(10/37): librgw2-20.3.0-5312.gabffb617.el9.x86_ 15 MB/s | 5.6 MB 00:00 2026-02-26T03:06:42.098 INFO:teuthology.orchestra.run.trial053.stdout:(11/37): python3-ceph-common-20.3.0-5312.gabffb 9.5 MB/s | 185 kB 00:00 2026-02-26T03:06:42.099 INFO:teuthology.orchestra.run.trial053.stdout:(12/37): python3-cephfs-20.3.0-5312.gabffb617.e 10 MB/s | 203 kB 00:00 2026-02-26T03:06:42.110 INFO:teuthology.orchestra.run.trial053.stdout:(13/37): python3-rados-20.3.0-5312.gabffb617.el 25 MB/s | 324 kB 00:00 2026-02-26T03:06:42.111 INFO:teuthology.orchestra.run.trial053.stdout:(14/37): python3-rbd-20.3.0-5312.gabffb617.el9. 23 MB/s | 305 kB 00:00 2026-02-26T03:06:42.112 INFO:teuthology.orchestra.run.trial053.stdout:(15/37): python3-rgw-20.3.0-5312.gabffb617.el9. 7.6 MB/s | 100 kB 00:00 2026-02-26T03:06:42.121 INFO:teuthology.orchestra.run.trial053.stdout:(16/37): mailcap-2.1.49-5.el9.noarch.rpm 3.0 MB/s | 33 kB 00:00 2026-02-26T03:06:42.126 INFO:teuthology.orchestra.run.trial053.stdout:(17/37): pciutils-3.7.0-7.el9.x86_64.rpm 6.7 MB/s | 93 kB 00:00 2026-02-26T03:06:42.140 INFO:teuthology.orchestra.run.trial053.stdout:(18/37): nvme-cli-2.16-1.el9.x86_64.rpm 41 MB/s | 1.2 MB 00:00 2026-02-26T03:06:42.456 INFO:teuthology.orchestra.run.trial053.stdout:[MIRROR] libnbd-1.20.3-4.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libnbd-1.20.3-4.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-26T03:06:42.466 INFO:teuthology.orchestra.run.trial053.stdout:[MIRROR] boost-program-options-1.75.0-13.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/boost-program-options-1.75.0-13.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-26T03:06:42.467 INFO:teuthology.orchestra.run.trial053.stdout:[MIRROR] libpmemobj-1.12.1-1.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libpmemobj-1.12.1-1.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-26T03:06:42.807 INFO:teuthology.orchestra.run.trial053.stdout:[MIRROR] boost-program-options-1.75.0-13.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/boost-program-options-1.75.0-13.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-26T03:06:42.808 INFO:teuthology.orchestra.run.trial053.stdout:[MIRROR] libnbd-1.20.3-4.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libnbd-1.20.3-4.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-26T03:06:42.809 INFO:teuthology.orchestra.run.trial053.stdout:[MIRROR] libpmemobj-1.12.1-1.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libpmemobj-1.12.1-1.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-26T03:06:43.204 INFO:teuthology.orchestra.run.trial053.stdout:(19/37): boost-program-options-1.75.0-13.el9.x8 96 kB/s | 104 kB 00:01 2026-02-26T03:06:43.259 INFO:teuthology.orchestra.run.trial053.stdout:(20/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 143 kB/s | 160 kB 00:01 2026-02-26T03:06:43.266 INFO:teuthology.orchestra.run.trial053.stdout:(21/37): libnbd-1.20.3-4.el9.x86_64.rpm 144 kB/s | 164 kB 00:01 2026-02-26T03:06:43.531 INFO:teuthology.orchestra.run.trial053.stdout:[MIRROR] librabbitmq-0.11.0-7.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librabbitmq-0.11.0-7.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-26T03:06:43.580 INFO:teuthology.orchestra.run.trial053.stdout:[MIRROR] librdkafka-1.6.1-102.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librdkafka-1.6.1-102.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-26T03:06:43.589 INFO:teuthology.orchestra.run.trial053.stdout:[MIRROR] lttng-ust-2.12.0-6.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/lttng-ust-2.12.0-6.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-26T03:06:43.631 INFO:teuthology.orchestra.run.trial053.stdout:(22/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 106 kB/s | 45 kB 00:00 2026-02-26T03:06:43.783 INFO:teuthology.orchestra.run.trial053.stdout:(23/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 565 kB/s | 292 kB 00:00 2026-02-26T03:06:43.833 INFO:teuthology.orchestra.run.trial053.stdout:(24/37): qatlib-25.08.0-2.el9.x86_64.rpm 1.2 MB/s | 240 kB 00:00 2026-02-26T03:06:43.852 INFO:teuthology.orchestra.run.trial053.stdout:(25/37): librdkafka-1.6.1-102.el9.x86_64.rpm 1.1 MB/s | 662 kB 00:00 2026-02-26T03:06:43.866 INFO:teuthology.orchestra.run.trial053.stdout:(26/37): qatlib-service-25.08.0-2.el9.x86_64.rp 450 kB/s | 37 kB 00:00 2026-02-26T03:06:43.922 INFO:teuthology.orchestra.run.trial053.stdout:(27/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 749 kB/s | 66 kB 00:00 2026-02-26T03:06:44.176 INFO:teuthology.orchestra.run.trial053.stdout:[MIRROR] gperftools-libs-2.9.1-3.el9.x86_64.rpm: Status code: 503 for https://ftp-osl.osuosl.org/pub/fedora-epel/9/Everything/x86_64/Packages/g/gperftools-libs-2.9.1-3.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-26T03:06:44.187 INFO:teuthology.orchestra.run.trial053.stdout:[MIRROR] libarrow-9.0.0-15.el9.x86_64.rpm: Status code: 503 for https://ftp-osl.osuosl.org/pub/fedora-epel/9/Everything/x86_64/Packages/l/libarrow-9.0.0-15.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-26T03:06:44.240 INFO:teuthology.orchestra.run.trial053.stdout:[MIRROR] libarrow-doc-9.0.0-15.el9.noarch.rpm: Status code: 503 for https://ftp-osl.osuosl.org/pub/fedora-epel/9/Everything/x86_64/Packages/l/libarrow-doc-9.0.0-15.el9.noarch.rpm (IP: 140.211.166.134) 2026-02-26T03:06:44.335 INFO:teuthology.orchestra.run.trial053.stdout:[MIRROR] gperftools-libs-2.9.1-3.el9.x86_64.rpm: Status code: 503 for http://ftp-osl.osuosl.org/pub/fedora-epel/9/Everything/x86_64/Packages/g/gperftools-libs-2.9.1-3.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-26T03:06:44.344 INFO:teuthology.orchestra.run.trial053.stdout:[MIRROR] libarrow-9.0.0-15.el9.x86_64.rpm: Status code: 503 for http://ftp-osl.osuosl.org/pub/fedora-epel/9/Everything/x86_64/Packages/l/libarrow-9.0.0-15.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-26T03:06:44.397 INFO:teuthology.orchestra.run.trial053.stdout:[MIRROR] libarrow-doc-9.0.0-15.el9.noarch.rpm: Status code: 503 for http://ftp-osl.osuosl.org/pub/fedora-epel/9/Everything/x86_64/Packages/l/libarrow-doc-9.0.0-15.el9.noarch.rpm (IP: 140.211.166.134) 2026-02-26T03:06:44.501 INFO:teuthology.orchestra.run.trial053.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 43 kB/s | 25 kB 00:00 2026-02-26T03:06:44.549 INFO:teuthology.orchestra.run.trial053.stdout:(29/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 442 kB/s | 308 kB 00:00 2026-02-26T03:06:44.662 INFO:teuthology.orchestra.run.trial053.stdout:[MIRROR] liboath-2.6.12-1.el9.x86_64.rpm: Status code: 503 for http://ftp-osl.osuosl.org/pub/fedora-epel/9/Everything/x86_64/Packages/l/liboath-2.6.12-1.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-26T03:06:44.705 INFO:teuthology.orchestra.run.trial053.stdout:(30/37): libarrow-9.0.0-15.el9.x86_64.rpm 5.3 MB/s | 4.4 MB 00:00 2026-02-26T03:06:44.810 INFO:teuthology.orchestra.run.trial053.stdout:(31/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 7.9 MB/s | 838 kB 00:00 2026-02-26T03:06:44.844 INFO:teuthology.orchestra.run.trial053.stdout:(32/37): re2-20211101-20.el9.x86_64.rpm 5.5 MB/s | 191 kB 00:00 2026-02-26T03:06:44.862 INFO:teuthology.orchestra.run.trial053.stdout:(33/37): libunwind-1.6.2-1.el9.x86_64.rpm 215 kB/s | 67 kB 00:00 2026-02-26T03:06:44.912 INFO:teuthology.orchestra.run.trial053.stdout:(34/37): thrift-0.15.0-4.el9.x86_64.rpm 23 MB/s | 1.6 MB 00:00 2026-02-26T03:06:45.138 INFO:teuthology.orchestra.run.trial053.stdout:(35/37): liboath-2.6.12-1.el9.x86_64.rpm 77 kB/s | 49 kB 00:00 2026-02-26T03:06:45.780 INFO:teuthology.orchestra.run.trial053.stdout:(36/37): librados2-20.3.0-5312.gabffb617.el9.x8 3.6 MB/s | 3.3 MB 00:00 2026-02-26T03:06:45.795 INFO:teuthology.orchestra.run.trial053.stdout:(37/37): librbd1-20.3.0-5312.gabffb617.el9.x86_ 3.0 MB/s | 2.7 MB 00:00 2026-02-26T03:06:45.796 INFO:teuthology.orchestra.run.trial053.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:06:45.796 INFO:teuthology.orchestra.run.trial053.stdout:Total 10 MB/s | 61 MB 00:06 2026-02-26T03:06:45.894 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction check 2026-02-26T03:06:45.912 INFO:teuthology.orchestra.run.trial053.stdout:Transaction check succeeded. 2026-02-26T03:06:45.912 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction test 2026-02-26T03:06:46.158 INFO:teuthology.orchestra.run.trial053.stdout:Transaction test succeeded. 2026-02-26T03:06:46.158 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction 2026-02-26T03:06:46.449 INFO:teuthology.orchestra.run.trial053.stdout: Preparing : 1/1 2026-02-26T03:06:46.588 INFO:teuthology.orchestra.run.trial053.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-26T03:06:46.592 INFO:teuthology.orchestra.run.trial053.stdout: Upgrading : librados2-2:20.3.0-5312.gabffb617.el9.x86_64 2/39 2026-02-26T03:06:46.642 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: librados2-2:20.3.0-5312.gabffb617.el9.x86_64 2/39 2026-02-26T03:06:46.643 INFO:teuthology.orchestra.run.trial053.stdout: Installing : libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 3/39 2026-02-26T03:06:46.656 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 3/39 2026-02-26T03:06:46.659 INFO:teuthology.orchestra.run.trial053.stdout: Installing : libcephfs-daemon-2:20.3.0-5312.gabffb617.el9.x86_6 4/39 2026-02-26T03:06:46.659 INFO:teuthology.orchestra.run.trial053.stdout: Installing : libcephfs-proxy2-2:20.3.0-5312.gabffb617.el9.x86_6 5/39 2026-02-26T03:06:46.686 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5312.gabffb617.el9.x86_6 5/39 2026-02-26T03:06:46.689 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 6/39 2026-02-26T03:06:46.696 INFO:teuthology.orchestra.run.trial053.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-26T03:06:46.699 INFO:teuthology.orchestra.run.trial053.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-26T03:06:46.702 INFO:teuthology.orchestra.run.trial053.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-26T03:06:46.706 INFO:teuthology.orchestra.run.trial053.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-26T03:06:46.717 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-ceph-argparse-2:20.3.0-5312.gabffb617.el9. 11/39 2026-02-26T03:06:46.729 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_64 12/39 2026-02-26T03:06:46.730 INFO:teuthology.orchestra.run.trial053.stdout: Installing : libradosstriper1-2:20.3.0-5312.gabffb617.el9.x86_6 13/39 2026-02-26T03:06:46.750 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5312.gabffb617.el9.x86_6 13/39 2026-02-26T03:06:46.754 INFO:teuthology.orchestra.run.trial053.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-26T03:06:46.785 INFO:teuthology.orchestra.run.trial053.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-26T03:06:46.788 INFO:teuthology.orchestra.run.trial053.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-26T03:06:46.796 INFO:teuthology.orchestra.run.trial053.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-26T03:06:46.798 INFO:teuthology.orchestra.run.trial053.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-26T03:06:46.803 INFO:teuthology.orchestra.run.trial053.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-26T03:06:46.917 INFO:teuthology.orchestra.run.trial053.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-26T03:06:46.918 INFO:teuthology.orchestra.run.trial053.stdout: Upgrading : librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 21/39 2026-02-26T03:06:46.943 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 21/39 2026-02-26T03:06:46.945 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 22/39 2026-02-26T03:06:46.951 INFO:teuthology.orchestra.run.trial053.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-26T03:06:47.200 INFO:teuthology.orchestra.run.trial053.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-26T03:06:47.201 INFO:teuthology.orchestra.run.trial053.stdout: Installing : librgw2-2:20.3.0-5312.gabffb617.el9.x86_64 25/39 2026-02-26T03:06:47.217 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: librgw2-2:20.3.0-5312.gabffb617.el9.x86_64 25/39 2026-02-26T03:06:47.221 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 26/39 2026-02-26T03:06:47.225 INFO:teuthology.orchestra.run.trial053.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-26T03:06:47.227 INFO:teuthology.orchestra.run.trial053.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-26T03:06:47.228 INFO:teuthology.orchestra.run.trial053.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-26T03:06:47.246 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-26T03:06:47.362 INFO:teuthology.orchestra.run.trial053.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-26T03:06:47.364 INFO:teuthology.orchestra.run.trial053.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-26T03:06:47.643 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-26T03:06:47.644 INFO:teuthology.orchestra.run.trial053.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-26T03:06:47.644 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:06:47.687 INFO:teuthology.orchestra.run.trial053.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-26T03:06:47.689 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-ceph-common-2:20.3.0-5312.gabffb617.el9.x8 33/39 2026-02-26T03:06:48.631 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 34/39 2026-02-26T03:06:48.634 INFO:teuthology.orchestra.run.trial053.stdout: Installing : ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 34/39 2026-02-26T03:06:48.894 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 34/39 2026-02-26T03:06:48.897 INFO:teuthology.orchestra.run.trial053.stdout: Installing : ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 35/39 2026-02-26T03:06:48.936 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 35/39 2026-02-26T03:06:48.936 INFO:teuthology.orchestra.run.trial053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-26T03:06:48.936 INFO:teuthology.orchestra.run.trial053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-26T03:06:48.936 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:06:48.938 INFO:teuthology.orchestra.run.trial053.stdout: Installing : ceph-selinux-2:20.3.0-5312.gabffb617.el9.x86_64 36/39 2026-02-26T03:06:53.436 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5312.gabffb617.el9.x86_64 36/39 2026-02-26T03:06:53.436 INFO:teuthology.orchestra.run.trial053.stdout:skipping the directory /sys 2026-02-26T03:06:53.436 INFO:teuthology.orchestra.run.trial053.stdout:skipping the directory /proc 2026-02-26T03:06:53.436 INFO:teuthology.orchestra.run.trial053.stdout:skipping the directory /mnt 2026-02-26T03:06:53.436 INFO:teuthology.orchestra.run.trial053.stdout:skipping the directory /var/tmp 2026-02-26T03:06:53.436 INFO:teuthology.orchestra.run.trial053.stdout:skipping the directory /home 2026-02-26T03:06:53.436 INFO:teuthology.orchestra.run.trial053.stdout:skipping the directory /root 2026-02-26T03:06:53.437 INFO:teuthology.orchestra.run.trial053.stdout:skipping the directory /tmp 2026-02-26T03:06:53.437 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:06:53.927 INFO:teuthology.orchestra.run.trial053.stdout: Installing : ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 37/39 2026-02-26T03:06:53.951 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 37/39 2026-02-26T03:06:53.951 INFO:teuthology.orchestra.run.trial053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:06:53.951 INFO:teuthology.orchestra.run.trial053.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-26T03:06:53.951 INFO:teuthology.orchestra.run.trial053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-26T03:06:53.951 INFO:teuthology.orchestra.run.trial053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-26T03:06:53.952 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:06:53.953 INFO:teuthology.orchestra.run.trial053.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-26T03:06:53.965 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-26T03:06:53.965 INFO:teuthology.orchestra.run.trial053.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-26T03:06:54.022 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-26T03:06:54.022 INFO:teuthology.orchestra.run.trial053.stdout:Creating group 'qat' with GID 992. 2026-02-26T03:06:54.022 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:06:54.327 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 1/39 2026-02-26T03:06:54.327 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 2/39 2026-02-26T03:06:54.327 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 3/39 2026-02-26T03:06:54.327 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : ceph-selinux-2:20.3.0-5312.gabffb617.el9.x86_64 4/39 2026-02-26T03:06:54.328 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : libcephfs-daemon-2:20.3.0-5312.gabffb617.el9.x86_6 5/39 2026-02-26T03:06:54.328 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5312.gabffb617.el9.x86_6 6/39 2026-02-26T03:06:54.328 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 7/39 2026-02-26T03:06:54.328 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : libradosstriper1-2:20.3.0-5312.gabffb617.el9.x86_6 8/39 2026-02-26T03:06:54.328 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : librgw2-2:20.3.0-5312.gabffb617.el9.x86_64 9/39 2026-02-26T03:06:54.328 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5312.gabffb617.el9. 10/39 2026-02-26T03:06:54.328 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-ceph-common-2:20.3.0-5312.gabffb617.el9.x8 11/39 2026-02-26T03:06:54.328 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_64 12/39 2026-02-26T03:06:54.328 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 13/39 2026-02-26T03:06:54.328 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 14/39 2026-02-26T03:06:54.328 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 15/39 2026-02-26T03:06:54.328 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-26T03:06:54.329 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 17/39 2026-02-26T03:06:54.329 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-26T03:06:54.329 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-26T03:06:54.329 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-26T03:06:54.329 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-26T03:06:54.329 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-26T03:06:54.329 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-26T03:06:54.329 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-26T03:06:54.329 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-26T03:06:54.329 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-26T03:06:54.329 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-26T03:06:54.329 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-26T03:06:54.329 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-26T03:06:54.330 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-26T03:06:54.330 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-26T03:06:54.330 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-26T03:06:54.330 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-26T03:06:54.330 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-26T03:06:54.330 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-26T03:06:54.330 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : librados2-2:20.3.0-5312.gabffb617.el9.x86_64 36/39 2026-02-26T03:06:54.330 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-26T03:06:54.330 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 38/39 2026-02-26T03:06:54.393 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-26T03:06:54.394 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:06:54.394 INFO:teuthology.orchestra.run.trial053.stdout:Upgraded: 2026-02-26T03:06:54.394 INFO:teuthology.orchestra.run.trial053.stdout: librados2-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:06:54.394 INFO:teuthology.orchestra.run.trial053.stdout: librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:06:54.394 INFO:teuthology.orchestra.run.trial053.stdout:Installed: 2026-02-26T03:06:54.394 INFO:teuthology.orchestra.run.trial053.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-26T03:06:54.394 INFO:teuthology.orchestra.run.trial053.stdout: ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:06:54.394 INFO:teuthology.orchestra.run.trial053.stdout: ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:06:54.394 INFO:teuthology.orchestra.run.trial053.stdout: ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:06:54.394 INFO:teuthology.orchestra.run.trial053.stdout: ceph-selinux-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:06:54.394 INFO:teuthology.orchestra.run.trial053.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-26T03:06:54.394 INFO:teuthology.orchestra.run.trial053.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-26T03:06:54.394 INFO:teuthology.orchestra.run.trial053.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-26T03:06:54.394 INFO:teuthology.orchestra.run.trial053.stdout: libcephfs-daemon-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:06:54.395 INFO:teuthology.orchestra.run.trial053.stdout: libcephfs-proxy2-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:06:54.395 INFO:teuthology.orchestra.run.trial053.stdout: libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:06:54.395 INFO:teuthology.orchestra.run.trial053.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-26T03:06:54.395 INFO:teuthology.orchestra.run.trial053.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-26T03:06:54.395 INFO:teuthology.orchestra.run.trial053.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-26T03:06:54.395 INFO:teuthology.orchestra.run.trial053.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-26T03:06:54.395 INFO:teuthology.orchestra.run.trial053.stdout: libradosstriper1-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:06:54.395 INFO:teuthology.orchestra.run.trial053.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-26T03:06:54.395 INFO:teuthology.orchestra.run.trial053.stdout: librgw2-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:06:54.395 INFO:teuthology.orchestra.run.trial053.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-26T03:06:54.395 INFO:teuthology.orchestra.run.trial053.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-26T03:06:54.395 INFO:teuthology.orchestra.run.trial053.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-26T03:06:54.396 INFO:teuthology.orchestra.run.trial053.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-26T03:06:54.396 INFO:teuthology.orchestra.run.trial053.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-26T03:06:54.396 INFO:teuthology.orchestra.run.trial053.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-26T03:06:54.396 INFO:teuthology.orchestra.run.trial053.stdout: python3-ceph-argparse-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:06:54.396 INFO:teuthology.orchestra.run.trial053.stdout: python3-ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:06:54.396 INFO:teuthology.orchestra.run.trial053.stdout: python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:06:54.396 INFO:teuthology.orchestra.run.trial053.stdout: python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:06:54.396 INFO:teuthology.orchestra.run.trial053.stdout: python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:06:54.396 INFO:teuthology.orchestra.run.trial053.stdout: python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:06:54.396 INFO:teuthology.orchestra.run.trial053.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-26T03:06:54.396 INFO:teuthology.orchestra.run.trial053.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-26T03:06:54.396 INFO:teuthology.orchestra.run.trial053.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-26T03:06:54.396 INFO:teuthology.orchestra.run.trial053.stdout: re2-1:20211101-20.el9.x86_64 2026-02-26T03:06:54.397 INFO:teuthology.orchestra.run.trial053.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-26T03:06:54.397 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:06:54.397 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:06:54.435 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install ceph-test 2026-02-26T03:06:54.753 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:00:16 ago on Thu 26 Feb 2026 03:06:38 AM UTC. 2026-02-26T03:06:54.837 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:06:54.838 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:06:54.838 INFO:teuthology.orchestra.run.trial053.stdout: Package Arch Version Repository Size 2026-02-26T03:06:54.839 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:06:54.839 INFO:teuthology.orchestra.run.trial053.stdout:Installing: 2026-02-26T03:06:54.839 INFO:teuthology.orchestra.run.trial053.stdout: ceph-test x86_64 2:20.3.0-5312.gabffb617.el9 ceph 126 M 2026-02-26T03:06:54.839 INFO:teuthology.orchestra.run.trial053.stdout:Installing dependencies: 2026-02-26T03:06:54.839 INFO:teuthology.orchestra.run.trial053.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-26T03:06:54.839 INFO:teuthology.orchestra.run.trial053.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-26T03:06:54.839 INFO:teuthology.orchestra.run.trial053.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-26T03:06:54.839 INFO:teuthology.orchestra.run.trial053.stdout: libcephsqlite x86_64 2:20.3.0-5312.gabffb617.el9 ceph 153 k 2026-02-26T03:06:54.839 INFO:teuthology.orchestra.run.trial053.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-26T03:06:54.840 INFO:teuthology.orchestra.run.trial053.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-26T03:06:54.840 INFO:teuthology.orchestra.run.trial053.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-26T03:06:54.840 INFO:teuthology.orchestra.run.trial053.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-26T03:06:54.840 INFO:teuthology.orchestra.run.trial053.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-26T03:06:54.840 INFO:teuthology.orchestra.run.trial053.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-26T03:06:54.840 INFO:teuthology.orchestra.run.trial053.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-26T03:06:54.840 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:06:54.840 INFO:teuthology.orchestra.run.trial053.stdout:Transaction Summary 2026-02-26T03:06:54.840 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:06:54.840 INFO:teuthology.orchestra.run.trial053.stdout:Install 12 Packages 2026-02-26T03:06:54.840 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:06:54.841 INFO:teuthology.orchestra.run.trial053.stdout:Total download size: 131 M 2026-02-26T03:06:54.841 INFO:teuthology.orchestra.run.trial053.stdout:Installed size: 976 M 2026-02-26T03:06:54.841 INFO:teuthology.orchestra.run.trial053.stdout:Downloading Packages: 2026-02-26T03:06:55.420 INFO:teuthology.orchestra.run.trial053.stdout:(1/12): libcephsqlite-20.3.0-5312.gabffb617.el9 1.5 MB/s | 153 kB 00:00 2026-02-26T03:06:55.480 INFO:teuthology.orchestra.run.trial053.stdout:(2/12): libatomic-11.5.0-14.el9.x86_64.rpm 388 kB/s | 23 kB 00:00 2026-02-26T03:06:55.520 INFO:teuthology.orchestra.run.trial053.stdout:(3/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 2.2 MB/s | 94 kB 00:00 2026-02-26T03:06:55.559 INFO:teuthology.orchestra.run.trial053.stdout:(4/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 8.7 MB/s | 2.1 MB 00:00 2026-02-26T03:06:55.580 INFO:teuthology.orchestra.run.trial053.stdout:(5/12): libasan8-15.2.1-7.el9.x86_64.rpm 8.7 MB/s | 525 kB 00:00 2026-02-26T03:06:55.591 INFO:teuthology.orchestra.run.trial053.stdout:(6/12): libxslt-1.1.34-12.el9.x86_64.rpm 20 MB/s | 233 kB 00:00 2026-02-26T03:06:55.603 INFO:teuthology.orchestra.run.trial053.stdout:(7/12): libubsan-11.5.0-14.el9.x86_64.rpm 3.9 MB/s | 176 kB 00:00 2026-02-26T03:06:55.613 INFO:teuthology.orchestra.run.trial053.stdout:(8/12): protobuf-3.14.0-17.el9.x86_64.rpm 45 MB/s | 1.0 MB 00:00 2026-02-26T03:06:55.623 INFO:teuthology.orchestra.run.trial053.stdout:(9/12): socat-1.7.4.1-8.el9.x86_64.rpm 14 MB/s | 303 kB 00:00 2026-02-26T03:06:55.624 INFO:teuthology.orchestra.run.trial053.stdout:(10/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 5.9 MB/s | 64 kB 00:00 2026-02-26T03:06:55.645 INFO:teuthology.orchestra.run.trial053.stdout:(11/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 5.5 MB/s | 122 kB 00:00 2026-02-26T03:06:58.145 INFO:teuthology.orchestra.run.trial053.stdout:(12/12): ceph-test-20.3.0-5312.gabffb617.el9.x8 45 MB/s | 126 MB 00:02 2026-02-26T03:06:58.147 INFO:teuthology.orchestra.run.trial053.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:06:58.147 INFO:teuthology.orchestra.run.trial053.stdout:Total 40 MB/s | 131 MB 00:03 2026-02-26T03:06:58.231 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction check 2026-02-26T03:06:58.242 INFO:teuthology.orchestra.run.trial053.stdout:Transaction check succeeded. 2026-02-26T03:06:58.242 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction test 2026-02-26T03:06:58.667 INFO:teuthology.orchestra.run.trial053.stdout:Transaction test succeeded. 2026-02-26T03:06:58.667 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction 2026-02-26T03:06:59.122 INFO:teuthology.orchestra.run.trial053.stdout: Preparing : 1/1 2026-02-26T03:06:59.131 INFO:teuthology.orchestra.run.trial053.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-26T03:06:59.140 INFO:teuthology.orchestra.run.trial053.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-26T03:06:59.145 INFO:teuthology.orchestra.run.trial053.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-26T03:06:59.149 INFO:teuthology.orchestra.run.trial053.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-26T03:06:59.152 INFO:teuthology.orchestra.run.trial053.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-26T03:06:59.156 INFO:teuthology.orchestra.run.trial053.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-26T03:06:59.157 INFO:teuthology.orchestra.run.trial053.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-26T03:06:59.174 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-26T03:06:59.177 INFO:teuthology.orchestra.run.trial053.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-26T03:06:59.188 INFO:teuthology.orchestra.run.trial053.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-26T03:06:59.196 INFO:teuthology.orchestra.run.trial053.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-26T03:06:59.197 INFO:teuthology.orchestra.run.trial053.stdout: Installing : libcephsqlite-2:20.3.0-5312.gabffb617.el9.x86_64 11/12 2026-02-26T03:07:06.074 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5312.gabffb617.el9.x86_64 11/12 2026-02-26T03:07:06.093 INFO:teuthology.orchestra.run.trial053.stdout: Installing : ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 12/12 2026-02-26T03:07:06.844 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 12/12 2026-02-26T03:07:06.844 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 1/12 2026-02-26T03:07:06.844 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : libcephsqlite-2:20.3.0-5312.gabffb617.el9.x86_64 2/12 2026-02-26T03:07:06.844 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-26T03:07:06.844 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-26T03:07:06.844 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-26T03:07:06.845 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-26T03:07:06.845 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-26T03:07:06.845 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-26T03:07:06.845 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-26T03:07:06.845 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-26T03:07:06.845 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-26T03:07:06.903 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-26T03:07:06.903 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:06.903 INFO:teuthology.orchestra.run.trial053.stdout:Installed: 2026-02-26T03:07:06.903 INFO:teuthology.orchestra.run.trial053.stdout: ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:07:06.904 INFO:teuthology.orchestra.run.trial053.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-26T03:07:06.904 INFO:teuthology.orchestra.run.trial053.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-26T03:07:06.904 INFO:teuthology.orchestra.run.trial053.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-26T03:07:06.904 INFO:teuthology.orchestra.run.trial053.stdout: libcephsqlite-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:07:06.904 INFO:teuthology.orchestra.run.trial053.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-26T03:07:06.904 INFO:teuthology.orchestra.run.trial053.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-26T03:07:06.904 INFO:teuthology.orchestra.run.trial053.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-26T03:07:06.904 INFO:teuthology.orchestra.run.trial053.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-26T03:07:06.904 INFO:teuthology.orchestra.run.trial053.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-26T03:07:06.904 INFO:teuthology.orchestra.run.trial053.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-26T03:07:06.904 INFO:teuthology.orchestra.run.trial053.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-26T03:07:06.904 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:06.905 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:07:06.946 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install ceph 2026-02-26T03:07:07.262 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:00:29 ago on Thu 26 Feb 2026 03:06:38 AM UTC. 2026-02-26T03:07:07.349 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:07:07.352 INFO:teuthology.orchestra.run.trial053.stdout:======================================================================================= 2026-02-26T03:07:07.352 INFO:teuthology.orchestra.run.trial053.stdout: Package Arch Version Repository Size 2026-02-26T03:07:07.352 INFO:teuthology.orchestra.run.trial053.stdout:======================================================================================= 2026-02-26T03:07:07.352 INFO:teuthology.orchestra.run.trial053.stdout:Installing: 2026-02-26T03:07:07.352 INFO:teuthology.orchestra.run.trial053.stdout: ceph x86_64 2:20.3.0-5312.gabffb617.el9 ceph 6.5 k 2026-02-26T03:07:07.352 INFO:teuthology.orchestra.run.trial053.stdout:Installing dependencies: 2026-02-26T03:07:07.352 INFO:teuthology.orchestra.run.trial053.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-26T03:07:07.352 INFO:teuthology.orchestra.run.trial053.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 57 k 2026-02-26T03:07:07.353 INFO:teuthology.orchestra.run.trial053.stdout: ceph-mds x86_64 2:20.3.0-5312.gabffb617.el9 ceph 2.0 M 2026-02-26T03:07:07.353 INFO:teuthology.orchestra.run.trial053.stdout: ceph-mgr x86_64 2:20.3.0-5312.gabffb617.el9 ceph 937 k 2026-02-26T03:07:07.353 INFO:teuthology.orchestra.run.trial053.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 297 k 2026-02-26T03:07:07.353 INFO:teuthology.orchestra.run.trial053.stdout: ceph-mon x86_64 2:20.3.0-5312.gabffb617.el9 ceph 4.9 M 2026-02-26T03:07:07.353 INFO:teuthology.orchestra.run.trial053.stdout: ceph-osd x86_64 2:20.3.0-5312.gabffb617.el9 ceph 163 k 2026-02-26T03:07:07.353 INFO:teuthology.orchestra.run.trial053.stdout: ceph-osd-classic x86_64 2:20.3.0-5312.gabffb617.el9 ceph 15 M 2026-02-26T03:07:07.353 INFO:teuthology.orchestra.run.trial053.stdout: ceph-osd-crimson x86_64 2:20.3.0-5312.gabffb617.el9 ceph 87 M 2026-02-26T03:07:07.353 INFO:teuthology.orchestra.run.trial053.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 17 k 2026-02-26T03:07:07.353 INFO:teuthology.orchestra.run.trial053.stdout: cephadm noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 1.0 M 2026-02-26T03:07:07.353 INFO:teuthology.orchestra.run.trial053.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-26T03:07:07.353 INFO:teuthology.orchestra.run.trial053.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-26T03:07:07.354 INFO:teuthology.orchestra.run.trial053.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-26T03:07:07.354 INFO:teuthology.orchestra.run.trial053.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-26T03:07:07.354 INFO:teuthology.orchestra.run.trial053.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-26T03:07:07.354 INFO:teuthology.orchestra.run.trial053.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-26T03:07:07.354 INFO:teuthology.orchestra.run.trial053.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-26T03:07:07.354 INFO:teuthology.orchestra.run.trial053.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-26T03:07:07.354 INFO:teuthology.orchestra.run.trial053.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-26T03:07:07.354 INFO:teuthology.orchestra.run.trial053.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-26T03:07:07.354 INFO:teuthology.orchestra.run.trial053.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-26T03:07:07.354 INFO:teuthology.orchestra.run.trial053.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-26T03:07:07.354 INFO:teuthology.orchestra.run.trial053.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-26T03:07:07.354 INFO:teuthology.orchestra.run.trial053.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-26T03:07:07.354 INFO:teuthology.orchestra.run.trial053.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-26T03:07:07.355 INFO:teuthology.orchestra.run.trial053.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-26T03:07:07.355 INFO:teuthology.orchestra.run.trial053.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-26T03:07:07.355 INFO:teuthology.orchestra.run.trial053.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-26T03:07:07.355 INFO:teuthology.orchestra.run.trial053.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-26T03:07:07.355 INFO:teuthology.orchestra.run.trial053.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-26T03:07:07.355 INFO:teuthology.orchestra.run.trial053.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-26T03:07:07.355 INFO:teuthology.orchestra.run.trial053.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-26T03:07:07.355 INFO:teuthology.orchestra.run.trial053.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-26T03:07:07.355 INFO:teuthology.orchestra.run.trial053.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-26T03:07:07.355 INFO:teuthology.orchestra.run.trial053.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-26T03:07:07.355 INFO:teuthology.orchestra.run.trial053.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-26T03:07:07.355 INFO:teuthology.orchestra.run.trial053.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-26T03:07:07.356 INFO:teuthology.orchestra.run.trial053.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-26T03:07:07.356 INFO:teuthology.orchestra.run.trial053.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-26T03:07:07.356 INFO:teuthology.orchestra.run.trial053.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-26T03:07:07.356 INFO:teuthology.orchestra.run.trial053.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-26T03:07:07.356 INFO:teuthology.orchestra.run.trial053.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-26T03:07:07.356 INFO:teuthology.orchestra.run.trial053.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-26T03:07:07.356 INFO:teuthology.orchestra.run.trial053.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-26T03:07:07.356 INFO:teuthology.orchestra.run.trial053.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-26T03:07:07.356 INFO:teuthology.orchestra.run.trial053.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-26T03:07:07.356 INFO:teuthology.orchestra.run.trial053.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-26T03:07:07.356 INFO:teuthology.orchestra.run.trial053.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-26T03:07:07.357 INFO:teuthology.orchestra.run.trial053.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-26T03:07:07.357 INFO:teuthology.orchestra.run.trial053.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-26T03:07:07.357 INFO:teuthology.orchestra.run.trial053.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-26T03:07:07.357 INFO:teuthology.orchestra.run.trial053.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-26T03:07:07.357 INFO:teuthology.orchestra.run.trial053.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-26T03:07:07.357 INFO:teuthology.orchestra.run.trial053.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-26T03:07:07.357 INFO:teuthology.orchestra.run.trial053.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-26T03:07:07.357 INFO:teuthology.orchestra.run.trial053.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-26T03:07:07.357 INFO:teuthology.orchestra.run.trial053.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-26T03:07:07.357 INFO:teuthology.orchestra.run.trial053.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-26T03:07:07.357 INFO:teuthology.orchestra.run.trial053.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-26T03:07:07.358 INFO:teuthology.orchestra.run.trial053.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-26T03:07:07.358 INFO:teuthology.orchestra.run.trial053.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-26T03:07:07.358 INFO:teuthology.orchestra.run.trial053.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-26T03:07:07.358 INFO:teuthology.orchestra.run.trial053.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-26T03:07:07.358 INFO:teuthology.orchestra.run.trial053.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-26T03:07:07.358 INFO:teuthology.orchestra.run.trial053.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-02-26T03:07:07.358 INFO:teuthology.orchestra.run.trial053.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-02-26T03:07:07.358 INFO:teuthology.orchestra.run.trial053.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-26T03:07:07.358 INFO:teuthology.orchestra.run.trial053.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-26T03:07:07.358 INFO:teuthology.orchestra.run.trial053.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-26T03:07:07.358 INFO:teuthology.orchestra.run.trial053.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-26T03:07:07.358 INFO:teuthology.orchestra.run.trial053.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-26T03:07:07.359 INFO:teuthology.orchestra.run.trial053.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-26T03:07:07.359 INFO:teuthology.orchestra.run.trial053.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-26T03:07:07.359 INFO:teuthology.orchestra.run.trial053.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-26T03:07:07.359 INFO:teuthology.orchestra.run.trial053.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-26T03:07:07.359 INFO:teuthology.orchestra.run.trial053.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-26T03:07:07.359 INFO:teuthology.orchestra.run.trial053.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-26T03:07:07.359 INFO:teuthology.orchestra.run.trial053.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-26T03:07:07.359 INFO:teuthology.orchestra.run.trial053.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-26T03:07:07.359 INFO:teuthology.orchestra.run.trial053.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-26T03:07:07.359 INFO:teuthology.orchestra.run.trial053.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-26T03:07:07.359 INFO:teuthology.orchestra.run.trial053.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-26T03:07:07.360 INFO:teuthology.orchestra.run.trial053.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-26T03:07:07.360 INFO:teuthology.orchestra.run.trial053.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-26T03:07:07.360 INFO:teuthology.orchestra.run.trial053.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-26T03:07:07.360 INFO:teuthology.orchestra.run.trial053.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-26T03:07:07.360 INFO:teuthology.orchestra.run.trial053.stdout:Installing weak dependencies: 2026-02-26T03:07:07.360 INFO:teuthology.orchestra.run.trial053.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 184 k 2026-02-26T03:07:07.360 INFO:teuthology.orchestra.run.trial053.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 51 M 2026-02-26T03:07:07.360 INFO:teuthology.orchestra.run.trial053.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 7.4 M 2026-02-26T03:07:07.360 INFO:teuthology.orchestra.run.trial053.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 22 k 2026-02-26T03:07:07.360 INFO:teuthology.orchestra.run.trial053.stdout: ceph-mgr-rook noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 50 k 2026-02-26T03:07:07.360 INFO:teuthology.orchestra.run.trial053.stdout: ceph-volume noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 301 k 2026-02-26T03:07:07.360 INFO:teuthology.orchestra.run.trial053.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-26T03:07:07.361 INFO:teuthology.orchestra.run.trial053.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-26T03:07:07.361 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:07.361 INFO:teuthology.orchestra.run.trial053.stdout:Transaction Summary 2026-02-26T03:07:07.361 INFO:teuthology.orchestra.run.trial053.stdout:======================================================================================= 2026-02-26T03:07:07.361 INFO:teuthology.orchestra.run.trial053.stdout:Install 96 Packages 2026-02-26T03:07:07.361 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:07.362 INFO:teuthology.orchestra.run.trial053.stdout:Total download size: 225 M 2026-02-26T03:07:07.362 INFO:teuthology.orchestra.run.trial053.stdout:Installed size: 1.5 G 2026-02-26T03:07:07.362 INFO:teuthology.orchestra.run.trial053.stdout:Downloading Packages: 2026-02-26T03:07:07.888 INFO:teuthology.orchestra.run.trial053.stdout:(1/96): ceph-20.3.0-5312.gabffb617.el9.x86_64.r 28 kB/s | 6.5 kB 00:00 2026-02-26T03:07:08.247 INFO:teuthology.orchestra.run.trial053.stdout:(2/96): ceph-mgr-20.3.0-5312.gabffb617.el9.x86_ 1.6 MB/s | 937 kB 00:00 2026-02-26T03:07:08.314 INFO:teuthology.orchestra.run.trial053.stdout:(3/96): ceph-osd-20.3.0-5312.gabffb617.el9.x86_ 2.4 MB/s | 163 kB 00:00 2026-02-26T03:07:08.594 INFO:teuthology.orchestra.run.trial053.stdout:(4/96): ceph-mds-20.3.0-5312.gabffb617.el9.x86_ 2.2 MB/s | 2.0 MB 00:00 2026-02-26T03:07:10.470 INFO:teuthology.orchestra.run.trial053.stdout:(5/96): ceph-osd-classic-20.3.0-5312.gabffb617. 6.8 MB/s | 15 MB 00:02 2026-02-26T03:07:10.493 INFO:teuthology.orchestra.run.trial053.stdout:(6/96): ceph-mon-20.3.0-5312.gabffb617.el9.x86_ 1.9 MB/s | 4.9 MB 00:02 2026-02-26T03:07:10.494 INFO:teuthology.orchestra.run.trial053.stdout:(7/96): ceph-grafana-dashboards-20.3.0-5312.gab 2.3 MB/s | 57 kB 00:00 2026-02-26T03:07:10.545 INFO:teuthology.orchestra.run.trial053.stdout:(8/96): ceph-mgr-cephadm-20.3.0-5312.gabffb617. 3.5 MB/s | 184 kB 00:00 2026-02-26T03:07:12.462 INFO:teuthology.orchestra.run.trial053.stdout:(9/96): ceph-mgr-diskprediction-local-20.3.0-53 3.9 MB/s | 7.4 MB 00:01 2026-02-26T03:07:12.498 INFO:teuthology.orchestra.run.trial053.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5312.gabffb6 622 kB/s | 22 kB 00:00 2026-02-26T03:07:12.600 INFO:teuthology.orchestra.run.trial053.stdout:(11/96): ceph-mgr-modules-core-20.3.0-5312.gabf 2.8 MB/s | 297 kB 00:00 2026-02-26T03:07:12.651 INFO:teuthology.orchestra.run.trial053.stdout:(12/96): ceph-mgr-rook-20.3.0-5312.gabffb617.el 995 kB/s | 50 kB 00:00 2026-02-26T03:07:12.704 INFO:teuthology.orchestra.run.trial053.stdout:(13/96): ceph-prometheus-alerts-20.3.0-5312.gab 331 kB/s | 17 kB 00:00 2026-02-26T03:07:12.811 INFO:teuthology.orchestra.run.trial053.stdout:(14/96): ceph-volume-20.3.0-5312.gabffb617.el9. 2.7 MB/s | 301 kB 00:00 2026-02-26T03:07:13.213 INFO:teuthology.orchestra.run.trial053.stdout:(15/96): cephadm-20.3.0-5312.gabffb617.el9.noar 2.5 MB/s | 1.0 MB 00:00 2026-02-26T03:07:13.318 INFO:teuthology.orchestra.run.trial053.stdout:(16/96): cryptsetup-2.8.1-3.el9.x86_64.rpm 3.3 MB/s | 351 kB 00:00 2026-02-26T03:07:13.340 INFO:teuthology.orchestra.run.trial053.stdout:(17/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 1.8 MB/s | 40 kB 00:00 2026-02-26T03:07:13.362 INFO:teuthology.orchestra.run.trial053.stdout:(18/96): libconfig-1.7.2-9.el9.x86_64.rpm 3.2 MB/s | 72 kB 00:00 2026-02-26T03:07:13.405 INFO:teuthology.orchestra.run.trial053.stdout:(19/96): libgfortran-11.5.0-14.el9.x86_64.rpm 18 MB/s | 794 kB 00:00 2026-02-26T03:07:13.430 INFO:teuthology.orchestra.run.trial053.stdout:(20/96): libquadmath-11.5.0-14.el9.x86_64.rpm 7.4 MB/s | 184 kB 00:00 2026-02-26T03:07:13.454 INFO:teuthology.orchestra.run.trial053.stdout:(21/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 10 MB/s | 253 kB 00:00 2026-02-26T03:07:13.539 INFO:teuthology.orchestra.run.trial053.stdout:(22/96): python3-cryptography-36.0.1-5.el9.x86_ 15 MB/s | 1.2 MB 00:00 2026-02-26T03:07:13.562 INFO:teuthology.orchestra.run.trial053.stdout:(23/96): python3-ply-3.11-14.el9.noarch.rpm 4.5 MB/s | 106 kB 00:00 2026-02-26T03:07:13.585 INFO:teuthology.orchestra.run.trial053.stdout:(24/96): python3-pycparser-2.20-6.el9.noarch.rp 5.6 MB/s | 135 kB 00:00 2026-02-26T03:07:13.609 INFO:teuthology.orchestra.run.trial053.stdout:(25/96): python3-pyparsing-2.4.7-9.el9.noarch.r 6.2 MB/s | 150 kB 00:00 2026-02-26T03:07:13.633 INFO:teuthology.orchestra.run.trial053.stdout:(26/96): python3-requests-2.25.1-10.el9.noarch. 5.1 MB/s | 126 kB 00:00 2026-02-26T03:07:13.657 INFO:teuthology.orchestra.run.trial053.stdout:(27/96): python3-urllib3-1.26.5-7.el9.noarch.rp 8.9 MB/s | 218 kB 00:00 2026-02-26T03:07:13.682 INFO:teuthology.orchestra.run.trial053.stdout:(28/96): unzip-6.0-59.el9.x86_64.rpm 7.4 MB/s | 182 kB 00:00 2026-02-26T03:07:13.705 INFO:teuthology.orchestra.run.trial053.stdout:(29/96): zip-3.0-35.el9.x86_64.rpm 11 MB/s | 266 kB 00:00 2026-02-26T03:07:13.963 INFO:teuthology.orchestra.run.trial053.stdout:(30/96): flexiblas-3.0.4-9.el9.x86_64.rpm 115 kB/s | 30 kB 00:00 2026-02-26T03:07:14.559 INFO:teuthology.orchestra.run.trial053.stdout:(31/96): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 5.0 MB/s | 3.0 MB 00:00 2026-02-26T03:07:14.638 INFO:teuthology.orchestra.run.trial053.stdout:(32/96): flexiblas-openblas-openmp-3.0.4-9.el9. 190 kB/s | 15 kB 00:00 2026-02-26T03:07:14.719 INFO:teuthology.orchestra.run.trial053.stdout:(33/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 3.0 MB/s | 246 kB 00:00 2026-02-26T03:07:14.799 INFO:teuthology.orchestra.run.trial053.stdout:(34/96): lua-5.4.4-4.el9.x86_64.rpm 2.3 MB/s | 188 kB 00:00 2026-02-26T03:07:14.877 INFO:teuthology.orchestra.run.trial053.stdout:(35/96): openblas-0.3.29-1.el9.x86_64.rpm 541 kB/s | 42 kB 00:00 2026-02-26T03:07:15.082 INFO:teuthology.orchestra.run.trial053.stdout:(36/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 26 MB/s | 5.3 MB 00:00 2026-02-26T03:07:15.269 INFO:teuthology.orchestra.run.trial053.stdout:(37/96): python3-babel-2.9.1-2.el9.noarch.rpm 32 MB/s | 6.0 MB 00:00 2026-02-26T03:07:15.350 INFO:teuthology.orchestra.run.trial053.stdout:(38/96): python3-devel-3.9.25-3.el9.x86_64.rpm 3.0 MB/s | 244 kB 00:00 2026-02-26T03:07:15.430 INFO:teuthology.orchestra.run.trial053.stdout:(39/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 3.0 MB/s | 249 kB 00:00 2026-02-26T03:07:15.509 INFO:teuthology.orchestra.run.trial053.stdout:(40/96): python3-jmespath-1.0.1-1.el9.noarch.rp 604 kB/s | 48 kB 00:00 2026-02-26T03:07:15.588 INFO:teuthology.orchestra.run.trial053.stdout:(41/96): python3-libstoragemgmt-1.10.1-1.el9.x8 2.2 MB/s | 177 kB 00:00 2026-02-26T03:07:15.676 INFO:teuthology.orchestra.run.trial053.stdout:(42/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 14 MB/s | 1.2 MB 00:00 2026-02-26T03:07:15.755 INFO:teuthology.orchestra.run.trial053.stdout:(43/96): python3-markupsafe-1.1.1-12.el9.x86_64 443 kB/s | 35 kB 00:00 2026-02-26T03:07:15.941 INFO:teuthology.orchestra.run.trial053.stdout:(44/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 33 MB/s | 6.1 MB 00:00 2026-02-26T03:07:16.022 INFO:teuthology.orchestra.run.trial053.stdout:(45/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 5.3 MB/s | 442 kB 00:00 2026-02-26T03:07:16.101 INFO:teuthology.orchestra.run.trial053.stdout:(46/96): python3-packaging-20.9-5.el9.noarch.rp 976 kB/s | 77 kB 00:00 2026-02-26T03:07:16.182 INFO:teuthology.orchestra.run.trial053.stdout:(47/96): python3-protobuf-3.14.0-17.el9.noarch. 3.3 MB/s | 267 kB 00:00 2026-02-26T03:07:16.279 INFO:teuthology.orchestra.run.trial053.stdout:(48/96): ceph-mgr-dashboard-20.3.0-5312.gabffb6 8.8 MB/s | 51 MB 00:05 2026-02-26T03:07:16.280 INFO:teuthology.orchestra.run.trial053.stdout:(49/96): python3-pyasn1-0.4.8-7.el9.noarch.rpm 1.6 MB/s | 157 kB 00:00 2026-02-26T03:07:16.359 INFO:teuthology.orchestra.run.trial053.stdout:(50/96): python3-requests-oauthlib-1.3.0-12.el9 680 kB/s | 54 kB 00:00 2026-02-26T03:07:16.793 INFO:teuthology.orchestra.run.trial053.stdout:(51/96): python3-pyasn1-modules-0.4.8-7.el9.noa 540 kB/s | 277 kB 00:00 2026-02-26T03:07:16.879 INFO:teuthology.orchestra.run.trial053.stdout:(52/96): python3-toml-0.10.2-6.el9.noarch.rpm 482 kB/s | 42 kB 00:00 2026-02-26T03:07:16.970 INFO:teuthology.orchestra.run.trial053.stdout:(53/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 31 MB/s | 19 MB 00:00 2026-02-26T03:07:16.995 INFO:teuthology.orchestra.run.trial053.stdout:(54/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 1.6 MB/s | 189 kB 00:00 2026-02-26T03:07:17.021 INFO:teuthology.orchestra.run.trial053.stdout:(55/96): lua-devel-5.4.4-4.el9.x86_64.rpm 864 kB/s | 22 kB 00:00 2026-02-26T03:07:17.050 INFO:teuthology.orchestra.run.trial053.stdout:(56/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 1.1 MB/s | 90 kB 00:00 2026-02-26T03:07:17.064 INFO:teuthology.orchestra.run.trial053.stdout:(57/96): protobuf-compiler-3.14.0-17.el9.x86_64 19 MB/s | 862 kB 00:00 2026-02-26T03:07:17.119 INFO:teuthology.orchestra.run.trial053.stdout:(58/96): grpc-data-1.46.7-10.el9.noarch.rpm 357 kB/s | 19 kB 00:00 2026-02-26T03:07:17.186 INFO:teuthology.orchestra.run.trial053.stdout:(59/96): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2026-02-26T03:07:17.193 INFO:teuthology.orchestra.run.trial053.stdout:(60/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 3.8 MB/s | 551 kB 00:00 2026-02-26T03:07:17.212 INFO:teuthology.orchestra.run.trial053.stdout:(61/96): python3-autocommand-2.2.2-8.el9.noarch 1.5 MB/s | 29 kB 00:00 2026-02-26T03:07:17.231 INFO:teuthology.orchestra.run.trial053.stdout:(62/96): python3-backports-tarfile-1.2.0-1.el9. 3.1 MB/s | 60 kB 00:00 2026-02-26T03:07:17.245 INFO:teuthology.orchestra.run.trial053.stdout:(63/96): python3-asyncssh-2.13.2-5.el9.noarch.r 9.1 MB/s | 548 kB 00:00 2026-02-26T03:07:17.250 INFO:teuthology.orchestra.run.trial053.stdout:(64/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 2.2 MB/s | 43 kB 00:00 2026-02-26T03:07:17.265 INFO:teuthology.orchestra.run.trial053.stdout:(65/96): python3-cachetools-4.2.4-1.el9.noarch. 1.6 MB/s | 32 kB 00:00 2026-02-26T03:07:17.269 INFO:teuthology.orchestra.run.trial053.stdout:(66/96): python3-certifi-2023.05.07-4.el9.noarc 766 kB/s | 14 kB 00:00 2026-02-26T03:07:17.287 INFO:teuthology.orchestra.run.trial053.stdout:(67/96): python3-cheroot-10.0.1-4.el9.noarch.rp 7.9 MB/s | 173 kB 00:00 2026-02-26T03:07:17.294 INFO:teuthology.orchestra.run.trial053.stdout:(68/96): python3-cherrypy-18.6.1-2.el9.noarch.r 14 MB/s | 358 kB 00:00 2026-02-26T03:07:17.313 INFO:teuthology.orchestra.run.trial053.stdout:(69/96): python3-google-auth-2.45.0-1.el9.noarc 9.6 MB/s | 254 kB 00:00 2026-02-26T03:07:17.334 INFO:teuthology.orchestra.run.trial053.stdout:(70/96): python3-grpcio-tools-1.46.7-10.el9.x86 6.7 MB/s | 144 kB 00:00 2026-02-26T03:07:17.357 INFO:teuthology.orchestra.run.trial053.stdout:(71/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 33 MB/s | 2.0 MB 00:00 2026-02-26T03:07:17.358 INFO:teuthology.orchestra.run.trial053.stdout:(72/96): python3-influxdb-5.3.1-1.el9.noarch.rp 5.8 MB/s | 139 kB 00:00 2026-02-26T03:07:17.376 INFO:teuthology.orchestra.run.trial053.stdout:(73/96): python3-isodate-0.6.1-3.el9.noarch.rpm 2.8 MB/s | 56 kB 00:00 2026-02-26T03:07:17.378 INFO:teuthology.orchestra.run.trial053.stdout:(74/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 524 kB/s | 11 kB 00:00 2026-02-26T03:07:17.395 INFO:teuthology.orchestra.run.trial053.stdout:(75/96): python3-jaraco-classes-3.2.1-5.el9.noa 945 kB/s | 18 kB 00:00 2026-02-26T03:07:17.401 INFO:teuthology.orchestra.run.trial053.stdout:(76/96): python3-jaraco-collections-3.0.0-8.el9 1.0 MB/s | 23 kB 00:00 2026-02-26T03:07:17.414 INFO:teuthology.orchestra.run.trial053.stdout:(77/96): python3-jaraco-context-6.0.1-3.el9.noa 1.0 MB/s | 20 kB 00:00 2026-02-26T03:07:17.422 INFO:teuthology.orchestra.run.trial053.stdout:(78/96): python3-jaraco-functools-3.5.0-2.el9.n 905 kB/s | 19 kB 00:00 2026-02-26T03:07:17.433 INFO:teuthology.orchestra.run.trial053.stdout:(79/96): python3-jaraco-text-4.0.0-2.el9.noarch 1.4 MB/s | 26 kB 00:00 2026-02-26T03:07:17.453 INFO:teuthology.orchestra.run.trial053.stdout:(80/96): python3-more-itertools-8.12.0-2.el9.no 3.9 MB/s | 79 kB 00:00 2026-02-26T03:07:17.463 INFO:teuthology.orchestra.run.trial053.stdout:(81/96): python3-kubernetes-26.1.0-3.el9.noarch 25 MB/s | 1.0 MB 00:00 2026-02-26T03:07:17.472 INFO:teuthology.orchestra.run.trial053.stdout:(82/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 4.3 MB/s | 86 kB 00:00 2026-02-26T03:07:17.482 INFO:teuthology.orchestra.run.trial053.stdout:(83/96): python3-natsort-7.1.1-5.el9.noarch.rpm 2.9 MB/s | 58 kB 00:00 2026-02-26T03:07:17.491 INFO:teuthology.orchestra.run.trial053.stdout:(84/96): python3-portend-3.1.0-2.el9.noarch.rpm 876 kB/s | 16 kB 00:00 2026-02-26T03:07:17.503 INFO:teuthology.orchestra.run.trial053.stdout:(85/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 4.1 MB/s | 90 kB 00:00 2026-02-26T03:07:17.510 INFO:teuthology.orchestra.run.trial053.stdout:(86/96): python3-repoze-lru-0.7-16.el9.noarch.r 1.6 MB/s | 31 kB 00:00 2026-02-26T03:07:17.526 INFO:teuthology.orchestra.run.trial053.stdout:(87/96): python3-routes-2.5.1-5.el9.noarch.rpm 8.4 MB/s | 188 kB 00:00 2026-02-26T03:07:17.530 INFO:teuthology.orchestra.run.trial053.stdout:(88/96): python3-rsa-4.9-2.el9.noarch.rpm 3.0 MB/s | 59 kB 00:00 2026-02-26T03:07:17.548 INFO:teuthology.orchestra.run.trial053.stdout:(89/96): python3-saml-1.16.0-1.el9.noarch.rpm 5.5 MB/s | 125 kB 00:00 2026-02-26T03:07:17.549 INFO:teuthology.orchestra.run.trial053.stdout:(90/96): python3-tempora-5.0.0-2.el9.noarch.rpm 1.8 MB/s | 36 kB 00:00 2026-02-26T03:07:17.569 INFO:teuthology.orchestra.run.trial053.stdout:(91/96): python3-websocket-client-1.2.3-2.el9.n 4.4 MB/s | 90 kB 00:00 2026-02-26T03:07:17.569 INFO:teuthology.orchestra.run.trial053.stdout:(92/96): python3-typing-extensions-4.15.0-1.el9 4.0 MB/s | 86 kB 00:00 2026-02-26T03:07:17.588 INFO:teuthology.orchestra.run.trial053.stdout:(93/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 2.5 MB/s | 48 kB 00:00 2026-02-26T03:07:17.589 INFO:teuthology.orchestra.run.trial053.stdout:(94/96): python3-xmltodict-0.12.0-15.el9.noarch 1.1 MB/s | 22 kB 00:00 2026-02-26T03:07:17.608 INFO:teuthology.orchestra.run.trial053.stdout:(95/96): python3-zc-lockfile-2.0-10.el9.noarch. 1.0 MB/s | 20 kB 00:00 2026-02-26T03:07:21.694 INFO:teuthology.orchestra.run.trial053.stdout:(96/96): ceph-osd-crimson-20.3.0-5312.gabffb617 6.6 MB/s | 87 MB 00:13 2026-02-26T03:07:21.695 INFO:teuthology.orchestra.run.trial053.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:07:21.696 INFO:teuthology.orchestra.run.trial053.stdout:Total 16 MB/s | 225 MB 00:14 2026-02-26T03:07:22.108 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction check 2026-02-26T03:07:22.146 INFO:teuthology.orchestra.run.trial053.stdout:Transaction check succeeded. 2026-02-26T03:07:22.146 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction test 2026-02-26T03:07:22.939 INFO:teuthology.orchestra.run.trial053.stdout:Transaction test succeeded. 2026-02-26T03:07:22.939 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction 2026-02-26T03:07:23.759 INFO:teuthology.orchestra.run.trial053.stdout: Preparing : 1/1 2026-02-26T03:07:23.761 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-26T03:07:23.764 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-26T03:07:23.786 INFO:teuthology.orchestra.run.trial053.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-26T03:07:23.820 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 4/96 2026-02-26T03:07:23.826 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-26T03:07:23.828 INFO:teuthology.orchestra.run.trial053.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-26T03:07:23.833 INFO:teuthology.orchestra.run.trial053.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 7/96 2026-02-26T03:07:23.856 INFO:teuthology.orchestra.run.trial053.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-26T03:07:23.869 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-26T03:07:23.872 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-26T03:07:23.878 INFO:teuthology.orchestra.run.trial053.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-26T03:07:23.880 INFO:teuthology.orchestra.run.trial053.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-26T03:07:23.891 INFO:teuthology.orchestra.run.trial053.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-26T03:07:23.897 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-26T03:07:23.921 INFO:teuthology.orchestra.run.trial053.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-26T03:07:23.924 INFO:teuthology.orchestra.run.trial053.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-26T03:07:23.973 INFO:teuthology.orchestra.run.trial053.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-26T03:07:23.986 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 18/96 2026-02-26T03:07:23.988 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-26T03:07:23.993 INFO:teuthology.orchestra.run.trial053.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-26T03:07:23.998 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-26T03:07:24.003 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-26T03:07:24.006 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-26T03:07:24.020 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-26T03:07:24.024 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-26T03:07:24.030 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-26T03:07:24.041 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-26T03:07:24.049 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-26T03:07:24.056 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-26T03:07:24.075 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-26T03:07:24.087 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-26T03:07:24.102 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-26T03:07:24.106 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-26T03:07:24.114 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-26T03:07:24.153 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-26T03:07:24.474 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-26T03:07:24.484 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-26T03:07:24.487 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-26T03:07:24.493 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-26T03:07:24.495 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-26T03:07:24.521 INFO:teuthology.orchestra.run.trial053.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-26T03:07:24.564 INFO:teuthology.orchestra.run.trial053.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-26T03:07:24.573 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-26T03:07:24.578 INFO:teuthology.orchestra.run.trial053.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-26T03:07:24.582 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-26T03:07:24.587 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-26T03:07:24.591 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-26T03:07:24.598 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-26T03:07:24.601 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-26T03:07:24.629 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-26T03:07:24.638 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-26T03:07:24.644 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-26T03:07:24.651 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-26T03:07:24.684 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-26T03:07:24.907 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 55/96 2026-02-26T03:07:24.930 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-26T03:07:24.934 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-26T03:07:24.992 INFO:teuthology.orchestra.run.trial053.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-26T03:07:24.993 INFO:teuthology.orchestra.run.trial053.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-26T03:07:25.014 INFO:teuthology.orchestra.run.trial053.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 60/96 2026-02-26T03:07:25.335 INFO:teuthology.orchestra.run.trial053.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 61/96 2026-02-26T03:07:25.407 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-26T03:07:26.070 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-26T03:07:26.087 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-26T03:07:26.096 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-26T03:07:26.103 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-26T03:07:26.116 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-26T03:07:26.132 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-26T03:07:26.207 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-26T03:07:26.216 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-26T03:07:26.239 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-26T03:07:26.268 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-26T03:07:26.318 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-26T03:07:26.324 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-26T03:07:26.329 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-26T03:07:26.330 INFO:teuthology.orchestra.run.trial053.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-26T03:07:26.348 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-26T03:07:26.349 INFO:teuthology.orchestra.run.trial053.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-26T03:07:26.349 INFO:teuthology.orchestra.run.trial053.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-26T03:07:26.349 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:26.356 INFO:teuthology.orchestra.run.trial053.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-26T03:07:26.381 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-26T03:07:26.381 INFO:teuthology.orchestra.run.trial053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-26T03:07:26.382 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:26.417 INFO:teuthology.orchestra.run.trial053.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-26T03:07:27.041 INFO:teuthology.orchestra.run.trial053.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 79/96 2026-02-26T03:07:32.516 INFO:teuthology.orchestra.run.trial053.stdout: Installing : ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_6 80/96 2026-02-26T03:07:32.579 INFO:teuthology.orchestra.run.trial053.stdout: Installing : ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86_6 81/96 2026-02-26T03:07:32.581 INFO:teuthology.orchestra.run.trial053.stdout: Installing : ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch 82/96 2026-02-26T03:07:32.596 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch 82/96 2026-02-26T03:07:32.596 INFO:teuthology.orchestra.run.trial053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:07:32.596 INFO:teuthology.orchestra.run.trial053.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-26T03:07:32.596 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:32.607 INFO:teuthology.orchestra.run.trial053.stdout: Installing : ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 83/96 2026-02-26T03:07:32.632 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 83/96 2026-02-26T03:07:32.632 INFO:teuthology.orchestra.run.trial053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:07:32.632 INFO:teuthology.orchestra.run.trial053.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-26T03:07:32.632 INFO:teuthology.orchestra.run.trial053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-26T03:07:32.632 INFO:teuthology.orchestra.run.trial053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-26T03:07:32.633 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:32.693 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: cephadm-2:20.3.0-5312.gabffb617.el9.noarch 84/96 2026-02-26T03:07:32.695 INFO:teuthology.orchestra.run.trial053.stdout: Installing : cephadm-2:20.3.0-5312.gabffb617.el9.noarch 84/96 2026-02-26T03:07:32.705 INFO:teuthology.orchestra.run.trial053.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5312.gabffb617.el9 85/96 2026-02-26T03:07:32.730 INFO:teuthology.orchestra.run.trial053.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5312.gabffb617.el 86/96 2026-02-26T03:07:32.731 INFO:teuthology.orchestra.run.trial053.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarc 87/96 2026-02-26T03:07:35.631 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarc 87/96 2026-02-26T03:07:35.638 INFO:teuthology.orchestra.run.trial053.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noa 88/96 2026-02-26T03:07:36.106 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noa 88/96 2026-02-26T03:07:36.107 INFO:teuthology.orchestra.run.trial053.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb 89/96 2026-02-26T03:07:36.120 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb 89/96 2026-02-26T03:07:36.120 INFO:teuthology.orchestra.run.trial053.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5312.gabffb617.el9.noa 90/96 2026-02-26T03:07:36.183 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5312.gabffb617.el9.noa 90/96 2026-02-26T03:07:36.225 INFO:teuthology.orchestra.run.trial053.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5312.gabffb617.el9. 91/96 2026-02-26T03:07:36.227 INFO:teuthology.orchestra.run.trial053.stdout: Installing : ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 92/96 2026-02-26T03:07:36.248 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 92/96 2026-02-26T03:07:36.248 INFO:teuthology.orchestra.run.trial053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:07:36.248 INFO:teuthology.orchestra.run.trial053.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-26T03:07:36.249 INFO:teuthology.orchestra.run.trial053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-26T03:07:36.249 INFO:teuthology.orchestra.run.trial053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-26T03:07:36.249 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:36.260 INFO:teuthology.orchestra.run.trial053.stdout: Installing : ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch 93/96 2026-02-26T03:07:36.488 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch 93/96 2026-02-26T03:07:36.489 INFO:teuthology.orchestra.run.trial053.stdout: Installing : ceph-mon-2:20.3.0-5312.gabffb617.el9.x86_64 94/96 2026-02-26T03:07:36.511 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: ceph-mon-2:20.3.0-5312.gabffb617.el9.x86_64 94/96 2026-02-26T03:07:36.511 INFO:teuthology.orchestra.run.trial053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:07:36.511 INFO:teuthology.orchestra.run.trial053.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-26T03:07:36.511 INFO:teuthology.orchestra.run.trial053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-26T03:07:36.512 INFO:teuthology.orchestra.run.trial053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-26T03:07:36.512 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:36.598 INFO:teuthology.orchestra.run.trial053.stdout: Installing : ceph-mds-2:20.3.0-5312.gabffb617.el9.x86_64 95/96 2026-02-26T03:07:36.618 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: ceph-mds-2:20.3.0-5312.gabffb617.el9.x86_64 95/96 2026-02-26T03:07:36.618 INFO:teuthology.orchestra.run.trial053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:07:36.618 INFO:teuthology.orchestra.run.trial053.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-26T03:07:36.618 INFO:teuthology.orchestra.run.trial053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-26T03:07:36.618 INFO:teuthology.orchestra.run.trial053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-26T03:07:36.618 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:36.619 INFO:teuthology.orchestra.run.trial053.stdout: Installing : ceph-2:20.3.0-5312.gabffb617.el9.x86_64 96/96 2026-02-26T03:07:36.626 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_6 96/96 2026-02-26T03:07:36.650 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86_6 96/96 2026-02-26T03:07:38.021 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: ceph-2:20.3.0-5312.gabffb617.el9.x86_64 96/96 2026-02-26T03:07:38.021 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : ceph-2:20.3.0-5312.gabffb617.el9.x86_64 1/96 2026-02-26T03:07:38.021 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : ceph-mds-2:20.3.0-5312.gabffb617.el9.x86_64 2/96 2026-02-26T03:07:38.021 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 3/96 2026-02-26T03:07:38.021 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : ceph-mon-2:20.3.0-5312.gabffb617.el9.x86_64 4/96 2026-02-26T03:07:38.022 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 5/96 2026-02-26T03:07:38.022 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_6 6/96 2026-02-26T03:07:38.022 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86_6 7/96 2026-02-26T03:07:38.022 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5312.gabffb617.el 8/96 2026-02-26T03:07:38.022 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarc 9/96 2026-02-26T03:07:38.023 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noa 10/96 2026-02-26T03:07:38.023 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb 11/96 2026-02-26T03:07:38.023 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5312.gabffb617.el9.noa 12/96 2026-02-26T03:07:38.023 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5312.gabffb617.el9. 13/96 2026-02-26T03:07:38.023 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch 14/96 2026-02-26T03:07:38.023 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5312.gabffb617.el9 15/96 2026-02-26T03:07:38.023 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch 16/96 2026-02-26T03:07:38.024 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : cephadm-2:20.3.0-5312.gabffb617.el9.noarch 17/96 2026-02-26T03:07:38.024 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 18/96 2026-02-26T03:07:38.024 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-26T03:07:38.024 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-26T03:07:38.024 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-26T03:07:38.025 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-26T03:07:38.025 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-26T03:07:38.025 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-26T03:07:38.025 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-26T03:07:38.025 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-26T03:07:38.025 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-26T03:07:38.025 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-26T03:07:38.026 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-26T03:07:38.026 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-26T03:07:38.026 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-26T03:07:38.026 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 32/96 2026-02-26T03:07:38.026 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 33/96 2026-02-26T03:07:38.026 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 34/96 2026-02-26T03:07:38.027 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-26T03:07:38.027 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-26T03:07:38.027 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-26T03:07:38.027 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-26T03:07:38.027 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-26T03:07:38.027 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 40/96 2026-02-26T03:07:38.027 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-26T03:07:38.028 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-26T03:07:38.028 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-26T03:07:38.028 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-26T03:07:38.028 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-26T03:07:38.028 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-26T03:07:38.028 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-26T03:07:38.029 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-26T03:07:38.029 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-26T03:07:38.029 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 50/96 2026-02-26T03:07:38.029 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 51/96 2026-02-26T03:07:38.029 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-26T03:07:38.029 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-26T03:07:38.030 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-26T03:07:38.030 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-26T03:07:38.030 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-26T03:07:38.030 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-26T03:07:38.030 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-26T03:07:38.030 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-26T03:07:38.031 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-26T03:07:38.031 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-26T03:07:38.031 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-26T03:07:38.031 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-26T03:07:38.031 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-26T03:07:38.031 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-26T03:07:38.032 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-26T03:07:38.032 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-26T03:07:38.032 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-26T03:07:38.032 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-26T03:07:38.032 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-26T03:07:38.032 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-26T03:07:38.032 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-26T03:07:38.033 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-26T03:07:38.033 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-26T03:07:38.033 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-26T03:07:38.033 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-26T03:07:38.033 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-26T03:07:38.033 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-26T03:07:38.033 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-26T03:07:38.034 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-26T03:07:38.034 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-26T03:07:38.034 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-26T03:07:38.034 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-26T03:07:38.034 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-26T03:07:38.034 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-26T03:07:38.034 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-26T03:07:38.034 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-26T03:07:38.035 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-26T03:07:38.035 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-26T03:07:38.035 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-26T03:07:38.035 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-26T03:07:38.035 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-26T03:07:38.035 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-26T03:07:38.035 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-26T03:07:38.035 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-26T03:07:38.081 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-26T03:07:38.081 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:38.081 INFO:teuthology.orchestra.run.trial053.stdout:Installed: 2026-02-26T03:07:38.081 INFO:teuthology.orchestra.run.trial053.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-26T03:07:38.081 INFO:teuthology.orchestra.run.trial053.stdout: ceph-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:07:38.082 INFO:teuthology.orchestra.run.trial053.stdout: ceph-grafana-dashboards-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:07:38.082 INFO:teuthology.orchestra.run.trial053.stdout: ceph-mds-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:07:38.082 INFO:teuthology.orchestra.run.trial053.stdout: ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:07:38.082 INFO:teuthology.orchestra.run.trial053.stdout: ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:07:38.082 INFO:teuthology.orchestra.run.trial053.stdout: ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:07:38.082 INFO:teuthology.orchestra.run.trial053.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:07:38.082 INFO:teuthology.orchestra.run.trial053.stdout: ceph-mgr-k8sevents-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:07:38.082 INFO:teuthology.orchestra.run.trial053.stdout: ceph-mgr-modules-core-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:07:38.082 INFO:teuthology.orchestra.run.trial053.stdout: ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:07:38.083 INFO:teuthology.orchestra.run.trial053.stdout: ceph-mon-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:07:38.083 INFO:teuthology.orchestra.run.trial053.stdout: ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:07:38.083 INFO:teuthology.orchestra.run.trial053.stdout: ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:07:38.083 INFO:teuthology.orchestra.run.trial053.stdout: ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:07:38.083 INFO:teuthology.orchestra.run.trial053.stdout: ceph-prometheus-alerts-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:07:38.083 INFO:teuthology.orchestra.run.trial053.stdout: ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:07:38.083 INFO:teuthology.orchestra.run.trial053.stdout: cephadm-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:07:38.083 INFO:teuthology.orchestra.run.trial053.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-26T03:07:38.083 INFO:teuthology.orchestra.run.trial053.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-26T03:07:38.083 INFO:teuthology.orchestra.run.trial053.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-26T03:07:38.083 INFO:teuthology.orchestra.run.trial053.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-26T03:07:38.083 INFO:teuthology.orchestra.run.trial053.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-26T03:07:38.083 INFO:teuthology.orchestra.run.trial053.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-26T03:07:38.084 INFO:teuthology.orchestra.run.trial053.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-26T03:07:38.084 INFO:teuthology.orchestra.run.trial053.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-26T03:07:38.084 INFO:teuthology.orchestra.run.trial053.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-26T03:07:38.084 INFO:teuthology.orchestra.run.trial053.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-26T03:07:38.084 INFO:teuthology.orchestra.run.trial053.stdout: lua-5.4.4-4.el9.x86_64 2026-02-26T03:07:38.084 INFO:teuthology.orchestra.run.trial053.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-26T03:07:38.084 INFO:teuthology.orchestra.run.trial053.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-26T03:07:38.084 INFO:teuthology.orchestra.run.trial053.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-26T03:07:38.084 INFO:teuthology.orchestra.run.trial053.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-26T03:07:38.084 INFO:teuthology.orchestra.run.trial053.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-26T03:07:38.084 INFO:teuthology.orchestra.run.trial053.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-26T03:07:38.084 INFO:teuthology.orchestra.run.trial053.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-26T03:07:38.085 INFO:teuthology.orchestra.run.trial053.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-26T03:07:38.085 INFO:teuthology.orchestra.run.trial053.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-26T03:07:38.085 INFO:teuthology.orchestra.run.trial053.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-26T03:07:38.085 INFO:teuthology.orchestra.run.trial053.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-26T03:07:38.085 INFO:teuthology.orchestra.run.trial053.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-26T03:07:38.085 INFO:teuthology.orchestra.run.trial053.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-26T03:07:38.085 INFO:teuthology.orchestra.run.trial053.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-26T03:07:38.085 INFO:teuthology.orchestra.run.trial053.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-26T03:07:38.085 INFO:teuthology.orchestra.run.trial053.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-26T03:07:38.085 INFO:teuthology.orchestra.run.trial053.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-26T03:07:38.085 INFO:teuthology.orchestra.run.trial053.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-26T03:07:38.085 INFO:teuthology.orchestra.run.trial053.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-26T03:07:38.085 INFO:teuthology.orchestra.run.trial053.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-26T03:07:38.086 INFO:teuthology.orchestra.run.trial053.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-26T03:07:38.086 INFO:teuthology.orchestra.run.trial053.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-26T03:07:38.086 INFO:teuthology.orchestra.run.trial053.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-26T03:07:38.086 INFO:teuthology.orchestra.run.trial053.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-26T03:07:38.086 INFO:teuthology.orchestra.run.trial053.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-26T03:07:38.086 INFO:teuthology.orchestra.run.trial053.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-26T03:07:38.086 INFO:teuthology.orchestra.run.trial053.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-26T03:07:38.086 INFO:teuthology.orchestra.run.trial053.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-26T03:07:38.086 INFO:teuthology.orchestra.run.trial053.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-26T03:07:38.086 INFO:teuthology.orchestra.run.trial053.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-26T03:07:38.086 INFO:teuthology.orchestra.run.trial053.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-26T03:07:38.086 INFO:teuthology.orchestra.run.trial053.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-26T03:07:38.086 INFO:teuthology.orchestra.run.trial053.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-26T03:07:38.087 INFO:teuthology.orchestra.run.trial053.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-26T03:07:38.087 INFO:teuthology.orchestra.run.trial053.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-26T03:07:38.087 INFO:teuthology.orchestra.run.trial053.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-26T03:07:38.087 INFO:teuthology.orchestra.run.trial053.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-26T03:07:38.087 INFO:teuthology.orchestra.run.trial053.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-26T03:07:38.087 INFO:teuthology.orchestra.run.trial053.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-26T03:07:38.087 INFO:teuthology.orchestra.run.trial053.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-26T03:07:38.087 INFO:teuthology.orchestra.run.trial053.stdout: python3-ply-3.11-14.el9.noarch 2026-02-26T03:07:38.087 INFO:teuthology.orchestra.run.trial053.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-26T03:07:38.087 INFO:teuthology.orchestra.run.trial053.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-26T03:07:38.087 INFO:teuthology.orchestra.run.trial053.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-26T03:07:38.087 INFO:teuthology.orchestra.run.trial053.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-02-26T03:07:38.087 INFO:teuthology.orchestra.run.trial053.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-02-26T03:07:38.087 INFO:teuthology.orchestra.run.trial053.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-26T03:07:38.088 INFO:teuthology.orchestra.run.trial053.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-26T03:07:38.088 INFO:teuthology.orchestra.run.trial053.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-26T03:07:38.088 INFO:teuthology.orchestra.run.trial053.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-26T03:07:38.088 INFO:teuthology.orchestra.run.trial053.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-26T03:07:38.088 INFO:teuthology.orchestra.run.trial053.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-26T03:07:38.088 INFO:teuthology.orchestra.run.trial053.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-26T03:07:38.088 INFO:teuthology.orchestra.run.trial053.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-26T03:07:38.088 INFO:teuthology.orchestra.run.trial053.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-26T03:07:38.088 INFO:teuthology.orchestra.run.trial053.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-26T03:07:38.088 INFO:teuthology.orchestra.run.trial053.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-26T03:07:38.088 INFO:teuthology.orchestra.run.trial053.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-26T03:07:38.088 INFO:teuthology.orchestra.run.trial053.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-26T03:07:38.088 INFO:teuthology.orchestra.run.trial053.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-26T03:07:38.089 INFO:teuthology.orchestra.run.trial053.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-26T03:07:38.089 INFO:teuthology.orchestra.run.trial053.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-26T03:07:38.089 INFO:teuthology.orchestra.run.trial053.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-26T03:07:38.089 INFO:teuthology.orchestra.run.trial053.stdout: unzip-6.0-59.el9.x86_64 2026-02-26T03:07:38.089 INFO:teuthology.orchestra.run.trial053.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-26T03:07:38.089 INFO:teuthology.orchestra.run.trial053.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-26T03:07:38.089 INFO:teuthology.orchestra.run.trial053.stdout: zip-3.0-35.el9.x86_64 2026-02-26T03:07:38.089 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:38.089 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:07:38.149 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install ceph-base 2026-02-26T03:07:38.472 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:01:00 ago on Thu 26 Feb 2026 03:06:38 AM UTC. 2026-02-26T03:07:38.538 INFO:teuthology.orchestra.run.trial053.stdout:Package ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:07:38.557 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:07:38.557 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:07:38.557 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:07:38.579 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install cephadm 2026-02-26T03:07:38.889 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:01:00 ago on Thu 26 Feb 2026 03:06:38 AM UTC. 2026-02-26T03:07:38.955 INFO:teuthology.orchestra.run.trial053.stdout:Package cephadm-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:07:38.974 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:07:38.975 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:07:38.975 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:07:38.995 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install ceph-immutable-object-cache 2026-02-26T03:07:39.310 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:01:01 ago on Thu 26 Feb 2026 03:06:38 AM UTC. 2026-02-26T03:07:39.394 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:07:39.395 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:07:39.395 INFO:teuthology.orchestra.run.trial053.stdout: Package Arch Version Repo Size 2026-02-26T03:07:39.395 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:07:39.395 INFO:teuthology.orchestra.run.trial053.stdout:Installing: 2026-02-26T03:07:39.395 INFO:teuthology.orchestra.run.trial053.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5312.gabffb617.el9 ceph 149 k 2026-02-26T03:07:39.395 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:39.395 INFO:teuthology.orchestra.run.trial053.stdout:Transaction Summary 2026-02-26T03:07:39.396 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:07:39.396 INFO:teuthology.orchestra.run.trial053.stdout:Install 1 Package 2026-02-26T03:07:39.396 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:39.396 INFO:teuthology.orchestra.run.trial053.stdout:Total download size: 149 k 2026-02-26T03:07:39.396 INFO:teuthology.orchestra.run.trial053.stdout:Installed size: 455 k 2026-02-26T03:07:39.396 INFO:teuthology.orchestra.run.trial053.stdout:Downloading Packages: 2026-02-26T03:07:40.007 INFO:teuthology.orchestra.run.trial053.stdout:ceph-immutable-object-cache-20.3.0-5312.gabffb6 244 kB/s | 149 kB 00:00 2026-02-26T03:07:40.008 INFO:teuthology.orchestra.run.trial053.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:07:40.008 INFO:teuthology.orchestra.run.trial053.stdout:Total 243 kB/s | 149 kB 00:00 2026-02-26T03:07:40.008 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction check 2026-02-26T03:07:40.013 INFO:teuthology.orchestra.run.trial053.stdout:Transaction check succeeded. 2026-02-26T03:07:40.013 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction test 2026-02-26T03:07:40.044 INFO:teuthology.orchestra.run.trial053.stdout:Transaction test succeeded. 2026-02-26T03:07:40.044 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction 2026-02-26T03:07:40.102 INFO:teuthology.orchestra.run.trial053.stdout: Preparing : 1/1 2026-02-26T03:07:40.103 INFO:teuthology.orchestra.run.trial053.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5312.gabffb617. 1/1 2026-02-26T03:07:40.126 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5312.gabffb617. 1/1 2026-02-26T03:07:40.126 INFO:teuthology.orchestra.run.trial053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:07:40.126 INFO:teuthology.orchestra.run.trial053.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-26T03:07:40.126 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:40.427 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5312.gabffb617. 1/1 2026-02-26T03:07:40.427 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:40.427 INFO:teuthology.orchestra.run.trial053.stdout:Installed: 2026-02-26T03:07:40.427 INFO:teuthology.orchestra.run.trial053.stdout: ceph-immutable-object-cache-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:07:40.427 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:40.427 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:07:40.453 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install ceph-mgr 2026-02-26T03:07:40.771 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:01:02 ago on Thu 26 Feb 2026 03:06:38 AM UTC. 2026-02-26T03:07:40.837 INFO:teuthology.orchestra.run.trial053.stdout:Package ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:07:40.856 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:07:40.857 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:07:40.857 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:07:40.879 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install ceph-mgr-dashboard 2026-02-26T03:07:41.197 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:01:03 ago on Thu 26 Feb 2026 03:06:38 AM UTC. 2026-02-26T03:07:41.262 INFO:teuthology.orchestra.run.trial053.stdout:Package ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:07:41.281 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:07:41.282 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:07:41.282 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:07:41.303 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-26T03:07:41.614 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:01:03 ago on Thu 26 Feb 2026 03:06:38 AM UTC. 2026-02-26T03:07:41.680 INFO:teuthology.orchestra.run.trial053.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:07:41.698 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:07:41.699 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:07:41.699 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:07:41.720 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install ceph-mgr-rook 2026-02-26T03:07:42.033 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:01:04 ago on Thu 26 Feb 2026 03:06:38 AM UTC. 2026-02-26T03:07:42.098 INFO:teuthology.orchestra.run.trial053.stdout:Package ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:07:42.117 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:07:42.117 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:07:42.118 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:07:42.138 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install ceph-mgr-cephadm 2026-02-26T03:07:42.451 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:01:04 ago on Thu 26 Feb 2026 03:06:38 AM UTC. 2026-02-26T03:07:42.516 INFO:teuthology.orchestra.run.trial053.stdout:Package ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:07:42.535 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:07:42.536 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:07:42.536 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:07:42.557 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install ceph-osd 2026-02-26T03:07:42.871 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:01:04 ago on Thu 26 Feb 2026 03:06:38 AM UTC. 2026-02-26T03:07:42.937 INFO:teuthology.orchestra.run.trial053.stdout:Package ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:07:42.955 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:07:42.956 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:07:42.956 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:07:42.978 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install ceph-osd-classic 2026-02-26T03:07:43.296 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:01:05 ago on Thu 26 Feb 2026 03:06:38 AM UTC. 2026-02-26T03:07:43.362 INFO:teuthology.orchestra.run.trial053.stdout:Package ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:07:43.381 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:07:43.381 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:07:43.381 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:07:43.403 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install ceph-fuse 2026-02-26T03:07:43.717 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:01:05 ago on Thu 26 Feb 2026 03:06:38 AM UTC. 2026-02-26T03:07:43.802 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:07:43.803 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:07:43.803 INFO:teuthology.orchestra.run.trial053.stdout: Package Arch Version Repo Size 2026-02-26T03:07:43.803 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:07:43.803 INFO:teuthology.orchestra.run.trial053.stdout:Installing: 2026-02-26T03:07:43.803 INFO:teuthology.orchestra.run.trial053.stdout: ceph-fuse x86_64 2:20.3.0-5312.gabffb617.el9 ceph 914 k 2026-02-26T03:07:43.803 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:43.803 INFO:teuthology.orchestra.run.trial053.stdout:Transaction Summary 2026-02-26T03:07:43.803 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:07:43.803 INFO:teuthology.orchestra.run.trial053.stdout:Install 1 Package 2026-02-26T03:07:43.804 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:43.804 INFO:teuthology.orchestra.run.trial053.stdout:Total download size: 914 k 2026-02-26T03:07:43.804 INFO:teuthology.orchestra.run.trial053.stdout:Installed size: 2.9 M 2026-02-26T03:07:43.804 INFO:teuthology.orchestra.run.trial053.stdout:Downloading Packages: 2026-02-26T03:07:44.810 INFO:teuthology.orchestra.run.trial053.stdout:ceph-fuse-20.3.0-5312.gabffb617.el9.x86_64.rpm 908 kB/s | 914 kB 00:01 2026-02-26T03:07:44.810 INFO:teuthology.orchestra.run.trial053.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:07:44.810 INFO:teuthology.orchestra.run.trial053.stdout:Total 907 kB/s | 914 kB 00:01 2026-02-26T03:07:44.810 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction check 2026-02-26T03:07:44.815 INFO:teuthology.orchestra.run.trial053.stdout:Transaction check succeeded. 2026-02-26T03:07:44.815 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction test 2026-02-26T03:07:44.848 INFO:teuthology.orchestra.run.trial053.stdout:Transaction test succeeded. 2026-02-26T03:07:44.849 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction 2026-02-26T03:07:44.939 INFO:teuthology.orchestra.run.trial053.stdout: Preparing : 1/1 2026-02-26T03:07:44.957 INFO:teuthology.orchestra.run.trial053.stdout: Installing : ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:07:45.220 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:07:45.282 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:07:45.283 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:45.283 INFO:teuthology.orchestra.run.trial053.stdout:Installed: 2026-02-26T03:07:45.283 INFO:teuthology.orchestra.run.trial053.stdout: ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:07:45.283 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:45.283 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:07:45.310 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install ceph-volume 2026-02-26T03:07:45.631 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:01:07 ago on Thu 26 Feb 2026 03:06:38 AM UTC. 2026-02-26T03:07:45.697 INFO:teuthology.orchestra.run.trial053.stdout:Package ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:07:45.716 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:07:45.717 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:07:45.717 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:07:45.738 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install librados-devel 2026-02-26T03:07:46.061 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:01:08 ago on Thu 26 Feb 2026 03:06:38 AM UTC. 2026-02-26T03:07:46.146 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:07:46.147 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:07:46.147 INFO:teuthology.orchestra.run.trial053.stdout: Package Arch Version Repo Size 2026-02-26T03:07:46.147 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:07:46.147 INFO:teuthology.orchestra.run.trial053.stdout:Installing: 2026-02-26T03:07:46.147 INFO:teuthology.orchestra.run.trial053.stdout: librados-devel x86_64 2:20.3.0-5312.gabffb617.el9 ceph 124 k 2026-02-26T03:07:46.147 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:46.147 INFO:teuthology.orchestra.run.trial053.stdout:Transaction Summary 2026-02-26T03:07:46.147 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:07:46.147 INFO:teuthology.orchestra.run.trial053.stdout:Install 1 Package 2026-02-26T03:07:46.147 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:46.147 INFO:teuthology.orchestra.run.trial053.stdout:Total download size: 124 k 2026-02-26T03:07:46.147 INFO:teuthology.orchestra.run.trial053.stdout:Installed size: 434 k 2026-02-26T03:07:46.147 INFO:teuthology.orchestra.run.trial053.stdout:Downloading Packages: 2026-02-26T03:07:46.609 INFO:teuthology.orchestra.run.trial053.stdout:librados-devel-20.3.0-5312.gabffb617.el9.x86_64 268 kB/s | 124 kB 00:00 2026-02-26T03:07:46.609 INFO:teuthology.orchestra.run.trial053.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:07:46.609 INFO:teuthology.orchestra.run.trial053.stdout:Total 267 kB/s | 124 kB 00:00 2026-02-26T03:07:46.609 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction check 2026-02-26T03:07:46.612 INFO:teuthology.orchestra.run.trial053.stdout:Transaction check succeeded. 2026-02-26T03:07:46.612 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction test 2026-02-26T03:07:46.644 INFO:teuthology.orchestra.run.trial053.stdout:Transaction test succeeded. 2026-02-26T03:07:46.644 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction 2026-02-26T03:07:46.699 INFO:teuthology.orchestra.run.trial053.stdout: Preparing : 1/1 2026-02-26T03:07:46.717 INFO:teuthology.orchestra.run.trial053.stdout: Installing : librados-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:07:46.803 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: librados-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:07:46.865 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : librados-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:07:46.865 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:46.865 INFO:teuthology.orchestra.run.trial053.stdout:Installed: 2026-02-26T03:07:46.865 INFO:teuthology.orchestra.run.trial053.stdout: librados-devel-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:07:46.865 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:46.865 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:07:46.894 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install libcephfs2 2026-02-26T03:07:47.213 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:01:09 ago on Thu 26 Feb 2026 03:06:38 AM UTC. 2026-02-26T03:07:47.279 INFO:teuthology.orchestra.run.trial053.stdout:Package libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:07:47.298 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:07:47.299 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:07:47.299 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:07:47.321 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install libcephfs-devel 2026-02-26T03:07:47.637 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:01:09 ago on Thu 26 Feb 2026 03:06:38 AM UTC. 2026-02-26T03:07:47.721 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:07:47.721 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:07:47.721 INFO:teuthology.orchestra.run.trial053.stdout: Package Arch Version Repo Size 2026-02-26T03:07:47.721 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:07:47.721 INFO:teuthology.orchestra.run.trial053.stdout:Installing: 2026-02-26T03:07:47.721 INFO:teuthology.orchestra.run.trial053.stdout: libcephfs-devel x86_64 2:20.3.0-5312.gabffb617.el9 ceph 37 k 2026-02-26T03:07:47.721 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:47.721 INFO:teuthology.orchestra.run.trial053.stdout:Transaction Summary 2026-02-26T03:07:47.722 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:07:47.722 INFO:teuthology.orchestra.run.trial053.stdout:Install 1 Package 2026-02-26T03:07:47.722 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:47.722 INFO:teuthology.orchestra.run.trial053.stdout:Total download size: 37 k 2026-02-26T03:07:47.722 INFO:teuthology.orchestra.run.trial053.stdout:Installed size: 171 k 2026-02-26T03:07:47.722 INFO:teuthology.orchestra.run.trial053.stdout:Downloading Packages: 2026-02-26T03:07:48.033 INFO:teuthology.orchestra.run.trial053.stdout:libcephfs-devel-20.3.0-5312.gabffb617.el9.x86_6 120 kB/s | 37 kB 00:00 2026-02-26T03:07:48.034 INFO:teuthology.orchestra.run.trial053.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:07:48.034 INFO:teuthology.orchestra.run.trial053.stdout:Total 120 kB/s | 37 kB 00:00 2026-02-26T03:07:48.034 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction check 2026-02-26T03:07:48.036 INFO:teuthology.orchestra.run.trial053.stdout:Transaction check succeeded. 2026-02-26T03:07:48.036 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction test 2026-02-26T03:07:48.039 INFO:teuthology.orchestra.run.trial053.stdout:Transaction test succeeded. 2026-02-26T03:07:48.039 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction 2026-02-26T03:07:48.061 INFO:teuthology.orchestra.run.trial053.stdout: Preparing : 1/1 2026-02-26T03:07:48.078 INFO:teuthology.orchestra.run.trial053.stdout: Installing : libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:07:48.138 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:07:48.199 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:07:48.199 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:48.199 INFO:teuthology.orchestra.run.trial053.stdout:Installed: 2026-02-26T03:07:48.199 INFO:teuthology.orchestra.run.trial053.stdout: libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:07:48.199 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:48.199 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:07:48.227 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install librados2 2026-02-26T03:07:48.545 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:01:10 ago on Thu 26 Feb 2026 03:06:38 AM UTC. 2026-02-26T03:07:48.610 INFO:teuthology.orchestra.run.trial053.stdout:Package librados2-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:07:48.627 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:07:48.628 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:07:48.628 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:07:48.650 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install librbd1 2026-02-26T03:07:48.965 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:01:10 ago on Thu 26 Feb 2026 03:06:38 AM UTC. 2026-02-26T03:07:49.032 INFO:teuthology.orchestra.run.trial053.stdout:Package librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:07:49.049 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:07:49.050 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:07:49.050 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:07:49.072 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install python3-rados 2026-02-26T03:07:49.386 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:01:11 ago on Thu 26 Feb 2026 03:06:38 AM UTC. 2026-02-26T03:07:49.452 INFO:teuthology.orchestra.run.trial053.stdout:Package python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:07:49.469 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:07:49.470 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:07:49.470 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:07:49.492 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install python3-rgw 2026-02-26T03:07:49.808 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:01:11 ago on Thu 26 Feb 2026 03:06:38 AM UTC. 2026-02-26T03:07:49.873 INFO:teuthology.orchestra.run.trial053.stdout:Package python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:07:49.891 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:07:49.891 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:07:49.892 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:07:49.913 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install python3-cephfs 2026-02-26T03:07:50.237 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:01:12 ago on Thu 26 Feb 2026 03:06:38 AM UTC. 2026-02-26T03:07:50.302 INFO:teuthology.orchestra.run.trial053.stdout:Package python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:07:50.320 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:07:50.320 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:07:50.321 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:07:50.341 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install python3-rbd 2026-02-26T03:07:50.654 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:01:12 ago on Thu 26 Feb 2026 03:06:38 AM UTC. 2026-02-26T03:07:50.719 INFO:teuthology.orchestra.run.trial053.stdout:Package python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:07:50.736 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:07:50.737 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:07:50.737 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:07:50.759 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install rbd-fuse 2026-02-26T03:07:51.077 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:01:13 ago on Thu 26 Feb 2026 03:06:38 AM UTC. 2026-02-26T03:07:51.160 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:07:51.161 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:07:51.161 INFO:teuthology.orchestra.run.trial053.stdout: Package Architecture Version Repository Size 2026-02-26T03:07:51.162 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:07:51.162 INFO:teuthology.orchestra.run.trial053.stdout:Installing: 2026-02-26T03:07:51.162 INFO:teuthology.orchestra.run.trial053.stdout: rbd-fuse x86_64 2:20.3.0-5312.gabffb617.el9 ceph 87 k 2026-02-26T03:07:51.162 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:51.162 INFO:teuthology.orchestra.run.trial053.stdout:Transaction Summary 2026-02-26T03:07:51.162 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:07:51.162 INFO:teuthology.orchestra.run.trial053.stdout:Install 1 Package 2026-02-26T03:07:51.162 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:51.162 INFO:teuthology.orchestra.run.trial053.stdout:Total download size: 87 k 2026-02-26T03:07:51.162 INFO:teuthology.orchestra.run.trial053.stdout:Installed size: 236 k 2026-02-26T03:07:51.162 INFO:teuthology.orchestra.run.trial053.stdout:Downloading Packages: 2026-02-26T03:07:51.405 INFO:teuthology.orchestra.run.trial053.stdout:rbd-fuse-20.3.0-5312.gabffb617.el9.x86_64.rpm 357 kB/s | 87 kB 00:00 2026-02-26T03:07:51.405 INFO:teuthology.orchestra.run.trial053.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:07:51.405 INFO:teuthology.orchestra.run.trial053.stdout:Total 355 kB/s | 87 kB 00:00 2026-02-26T03:07:51.406 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction check 2026-02-26T03:07:51.409 INFO:teuthology.orchestra.run.trial053.stdout:Transaction check succeeded. 2026-02-26T03:07:51.410 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction test 2026-02-26T03:07:51.442 INFO:teuthology.orchestra.run.trial053.stdout:Transaction test succeeded. 2026-02-26T03:07:51.442 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction 2026-02-26T03:07:51.493 INFO:teuthology.orchestra.run.trial053.stdout: Preparing : 1/1 2026-02-26T03:07:51.510 INFO:teuthology.orchestra.run.trial053.stdout: Installing : rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:07:51.598 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:07:51.660 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:07:51.660 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:51.660 INFO:teuthology.orchestra.run.trial053.stdout:Installed: 2026-02-26T03:07:51.660 INFO:teuthology.orchestra.run.trial053.stdout: rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:07:51.660 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:51.660 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:07:51.689 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install rbd-mirror 2026-02-26T03:07:52.005 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:01:14 ago on Thu 26 Feb 2026 03:06:38 AM UTC. 2026-02-26T03:07:52.089 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:07:52.089 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:07:52.089 INFO:teuthology.orchestra.run.trial053.stdout: Package Arch Version Repo Size 2026-02-26T03:07:52.089 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:07:52.089 INFO:teuthology.orchestra.run.trial053.stdout:Installing: 2026-02-26T03:07:52.089 INFO:teuthology.orchestra.run.trial053.stdout: rbd-mirror x86_64 2:20.3.0-5312.gabffb617.el9 ceph 2.6 M 2026-02-26T03:07:52.089 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:52.089 INFO:teuthology.orchestra.run.trial053.stdout:Transaction Summary 2026-02-26T03:07:52.090 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:07:52.090 INFO:teuthology.orchestra.run.trial053.stdout:Install 1 Package 2026-02-26T03:07:52.090 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:52.090 INFO:teuthology.orchestra.run.trial053.stdout:Total download size: 2.6 M 2026-02-26T03:07:52.090 INFO:teuthology.orchestra.run.trial053.stdout:Installed size: 11 M 2026-02-26T03:07:52.090 INFO:teuthology.orchestra.run.trial053.stdout:Downloading Packages: 2026-02-26T03:07:52.596 INFO:teuthology.orchestra.run.trial053.stdout:rbd-mirror-20.3.0-5312.gabffb617.el9.x86_64.rpm 5.1 MB/s | 2.6 MB 00:00 2026-02-26T03:07:52.596 INFO:teuthology.orchestra.run.trial053.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:07:52.596 INFO:teuthology.orchestra.run.trial053.stdout:Total 5.1 MB/s | 2.6 MB 00:00 2026-02-26T03:07:52.596 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction check 2026-02-26T03:07:52.600 INFO:teuthology.orchestra.run.trial053.stdout:Transaction check succeeded. 2026-02-26T03:07:52.601 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction test 2026-02-26T03:07:52.639 INFO:teuthology.orchestra.run.trial053.stdout:Transaction test succeeded. 2026-02-26T03:07:52.639 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction 2026-02-26T03:07:52.801 INFO:teuthology.orchestra.run.trial053.stdout: Preparing : 1/1 2026-02-26T03:07:52.803 INFO:teuthology.orchestra.run.trial053.stdout: Installing : rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:07:52.825 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:07:52.825 INFO:teuthology.orchestra.run.trial053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:07:52.825 INFO:teuthology.orchestra.run.trial053.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-26T03:07:52.825 INFO:teuthology.orchestra.run.trial053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-26T03:07:52.825 INFO:teuthology.orchestra.run.trial053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-26T03:07:52.825 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:53.123 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:07:53.123 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:53.123 INFO:teuthology.orchestra.run.trial053.stdout:Installed: 2026-02-26T03:07:53.123 INFO:teuthology.orchestra.run.trial053.stdout: rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:07:53.123 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:53.123 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:07:53.152 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install rbd-nbd 2026-02-26T03:07:53.485 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:01:15 ago on Thu 26 Feb 2026 03:06:38 AM UTC. 2026-02-26T03:07:53.570 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:07:53.571 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:07:53.571 INFO:teuthology.orchestra.run.trial053.stdout: Package Architecture Version Repository Size 2026-02-26T03:07:53.571 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:07:53.571 INFO:teuthology.orchestra.run.trial053.stdout:Installing: 2026-02-26T03:07:53.571 INFO:teuthology.orchestra.run.trial053.stdout: rbd-nbd x86_64 2:20.3.0-5312.gabffb617.el9 ceph 166 k 2026-02-26T03:07:53.571 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:53.571 INFO:teuthology.orchestra.run.trial053.stdout:Transaction Summary 2026-02-26T03:07:53.571 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:07:53.571 INFO:teuthology.orchestra.run.trial053.stdout:Install 1 Package 2026-02-26T03:07:53.571 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:53.572 INFO:teuthology.orchestra.run.trial053.stdout:Total download size: 166 k 2026-02-26T03:07:53.572 INFO:teuthology.orchestra.run.trial053.stdout:Installed size: 496 k 2026-02-26T03:07:53.572 INFO:teuthology.orchestra.run.trial053.stdout:Downloading Packages: 2026-02-26T03:07:53.891 INFO:teuthology.orchestra.run.trial053.stdout:rbd-nbd-20.3.0-5312.gabffb617.el9.x86_64.rpm 519 kB/s | 166 kB 00:00 2026-02-26T03:07:53.892 INFO:teuthology.orchestra.run.trial053.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:07:53.892 INFO:teuthology.orchestra.run.trial053.stdout:Total 517 kB/s | 166 kB 00:00 2026-02-26T03:07:53.892 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction check 2026-02-26T03:07:53.896 INFO:teuthology.orchestra.run.trial053.stdout:Transaction check succeeded. 2026-02-26T03:07:53.896 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction test 2026-02-26T03:07:53.927 INFO:teuthology.orchestra.run.trial053.stdout:Transaction test succeeded. 2026-02-26T03:07:53.928 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction 2026-02-26T03:07:53.984 INFO:teuthology.orchestra.run.trial053.stdout: Preparing : 1/1 2026-02-26T03:07:54.002 INFO:teuthology.orchestra.run.trial053.stdout: Installing : rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:07:54.090 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:07:54.152 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:07:54.152 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:54.152 INFO:teuthology.orchestra.run.trial053.stdout:Installed: 2026-02-26T03:07:54.153 INFO:teuthology.orchestra.run.trial053.stdout: rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:07:54.153 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:54.153 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:07:54.180 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install ceph-osd-crimson 2026-02-26T03:07:54.504 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:01:16 ago on Thu 26 Feb 2026 03:06:38 AM UTC. 2026-02-26T03:07:54.569 INFO:teuthology.orchestra.run.trial053.stdout:Package ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:07:54.586 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:07:54.587 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:07:54.587 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:07:54.608 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-26T03:07:54.923 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:01:16 ago on Thu 26 Feb 2026 03:06:38 AM UTC. 2026-02-26T03:07:55.007 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:07:55.007 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:07:55.008 INFO:teuthology.orchestra.run.trial053.stdout: Package Arch Version Repo Size 2026-02-26T03:07:55.008 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:07:55.008 INFO:teuthology.orchestra.run.trial053.stdout:Installing: 2026-02-26T03:07:55.008 INFO:teuthology.orchestra.run.trial053.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5312.gabffb617.el9 ceph 429 M 2026-02-26T03:07:55.008 INFO:teuthology.orchestra.run.trial053.stdout:Installing weak dependencies: 2026-02-26T03:07:55.008 INFO:teuthology.orchestra.run.trial053.stdout: ceph-debugsource x86_64 2:20.3.0-5312.gabffb617.el9 ceph 12 M 2026-02-26T03:07:55.008 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:55.008 INFO:teuthology.orchestra.run.trial053.stdout:Transaction Summary 2026-02-26T03:07:55.008 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:07:55.008 INFO:teuthology.orchestra.run.trial053.stdout:Install 2 Packages 2026-02-26T03:07:55.008 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:07:55.008 INFO:teuthology.orchestra.run.trial053.stdout:Total download size: 441 M 2026-02-26T03:07:55.008 INFO:teuthology.orchestra.run.trial053.stdout:Installed size: 2.7 G 2026-02-26T03:07:55.008 INFO:teuthology.orchestra.run.trial053.stdout:Downloading Packages: 2026-02-26T03:07:56.408 INFO:teuthology.orchestra.run.trial053.stdout:(1/2): ceph-debugsource-20.3.0-5312.gabffb617.e 8.6 MB/s | 12 MB 00:01 2026-02-26T03:08:32.819 INFO:teuthology.orchestra.run.trial053.stdout:(2/2): ceph-osd-crimson-debuginfo-20.3.0-5312.g 11 MB/s | 429 MB 00:37 2026-02-26T03:08:32.820 INFO:teuthology.orchestra.run.trial053.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:08:32.820 INFO:teuthology.orchestra.run.trial053.stdout:Total 12 MB/s | 441 MB 00:37 2026-02-26T03:08:32.820 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction check 2026-02-26T03:08:32.825 INFO:teuthology.orchestra.run.trial053.stdout:Transaction check succeeded. 2026-02-26T03:08:32.825 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction test 2026-02-26T03:08:34.183 INFO:teuthology.orchestra.run.trial053.stdout:Transaction test succeeded. 2026-02-26T03:08:34.184 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction 2026-02-26T03:08:37.762 INFO:teuthology.orchestra.run.trial053.stdout: Preparing : 1/1 2026-02-26T03:08:59.868 INFO:teuthology.orchestra.run.trial053.stdout: Installing : ceph-debugsource-2:20.3.0-5312.gabffb617.el9.x86_64 1/2 2026-02-26T03:08:59.886 INFO:teuthology.orchestra.run.trial053.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5312.gabffb617.e 2/2 2026-02-26T03:09:01.901 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5312.gabffb617.e 2/2 2026-02-26T03:09:01.901 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : ceph-debugsource-2:20.3.0-5312.gabffb617.el9.x86_64 1/2 2026-02-26T03:09:01.964 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5312.gabffb617.e 2/2 2026-02-26T03:09:01.964 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:09:01.964 INFO:teuthology.orchestra.run.trial053.stdout:Installed: 2026-02-26T03:09:01.964 INFO:teuthology.orchestra.run.trial053.stdout: ceph-debugsource-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:09:01.964 INFO:teuthology.orchestra.run.trial053.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:09:01.964 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:09:01.964 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:09:02.039 DEBUG:teuthology.parallel:result is None 2026-02-26T03:09:02.040 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:09:02.205 DEBUG:teuthology.orchestra.run.trial053:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-26T03:09:02.226 INFO:teuthology.orchestra.run.trial053.stdout:20.3.0-5312.gabffb617.el9 2026-02-26T03:09:02.226 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5312.gabffb617.el9 2026-02-26T03:09:02.226 INFO:teuthology.task.install:The correct ceph version 20.3.0-5312.gabffb617 is installed. 2026-02-26T03:09:02.229 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-26T03:09:02.229 DEBUG:teuthology.orchestra.run.trial053:> set -ex 2026-02-26T03:09:02.229 DEBUG:teuthology.orchestra.run.trial053:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-26T03:09:02.291 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-26T03:09:02.291 DEBUG:teuthology.orchestra.run.trial053:> set -ex 2026-02-26T03:09:02.291 DEBUG:teuthology.orchestra.run.trial053:> sudo dd of=/usr/bin/daemon-helper 2026-02-26T03:09:02.356 DEBUG:teuthology.orchestra.run.trial053:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-26T03:09:02.420 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-26T03:09:02.420 DEBUG:teuthology.orchestra.run.trial053:> set -ex 2026-02-26T03:09:02.420 DEBUG:teuthology.orchestra.run.trial053:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-26T03:09:02.483 DEBUG:teuthology.orchestra.run.trial053:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-26T03:09:02.549 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-26T03:09:02.549 DEBUG:teuthology.orchestra.run.trial053:> set -ex 2026-02-26T03:09:02.549 DEBUG:teuthology.orchestra.run.trial053:> sudo dd of=/usr/bin/stdin-killer 2026-02-26T03:09:02.613 DEBUG:teuthology.orchestra.run.trial053:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-26T03:09:02.677 INFO:teuthology.run_tasks:Running task pexec... 2026-02-26T03:09:02.691 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-26T03:09:02.691 DEBUG:teuthology.orchestra.run.trial053:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-26T03:09:02.719 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial053.front.sepia.ceph.com 2026-02-26T03:09:02.719 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-26T03:09:02.772 DEBUG:teuthology.parallel:result is None 2026-02-26T03:09:02.772 INFO:teuthology.run_tasks:Running task install... 2026-02-26T03:09:02.786 DEBUG:teuthology.task.install:project ceph 2026-02-26T03:09:02.786 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}} 2026-02-26T03:09:02.786 DEBUG:teuthology.task.install:config {'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'} 2026-02-26T03:09:02.787 INFO:teuthology.task.install:Using flavor: debug 2026-02-26T03:09:02.795 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-26T03:09:02.795 INFO:teuthology.task.install:extra packages: [] 2026-02-26T03:09:02.796 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8', 'tag': None, 'wait_for_package': False} 2026-02-26T03:09:02.796 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:09:02.955 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/ 2026-02-26T03:09:02.955 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5312.gabffb617 2026-02-26T03:09:03.315 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-26T03:09:03.315 DEBUG:teuthology.orchestra.run.trial053:> set -ex 2026-02-26T03:09:03.315 DEBUG:teuthology.orchestra.run.trial053:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-26T03:09:03.339 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-26T03:09:03.340 DEBUG:teuthology.orchestra.run.trial053:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-26T03:09:03.405 DEBUG:teuthology.orchestra.run.trial053:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-26T03:09:03.468 DEBUG:teuthology.orchestra.run.trial053:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-26T03:09:03.524 INFO:teuthology.orchestra.run.trial053.stdout:check_obsoletes = 1 2026-02-26T03:09:03.534 DEBUG:teuthology.orchestra.run.trial053:> sudo yum clean all 2026-02-26T03:09:03.693 INFO:teuthology.orchestra.run.trial053.stdout:56 files removed 2026-02-26T03:09:03.708 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install ceph-radosgw 2026-02-26T03:09:04.490 INFO:teuthology.orchestra.run.trial053.stdout:ceph packages for x86_64 150 kB/s | 94 kB 00:00 2026-02-26T03:09:05.102 INFO:teuthology.orchestra.run.trial053.stdout:ceph noarch packages 42 kB/s | 24 kB 00:00 2026-02-26T03:09:05.659 INFO:teuthology.orchestra.run.trial053.stdout:ceph source packages 3.9 kB/s | 2.1 kB 00:00 2026-02-26T03:09:09.251 INFO:teuthology.orchestra.run.trial053.stdout:CentOS Stream 9 - BaseOS 2.5 MB/s | 8.9 MB 00:03 2026-02-26T03:11:04.915 INFO:teuthology.orchestra.run.trial053.stdout:CentOS Stream 9 - AppStream 240 kB/s | 27 MB 01:55 2026-02-26T03:11:16.348 INFO:teuthology.orchestra.run.trial053.stdout:CentOS Stream 9 - CRB 900 kB/s | 8.0 MB 00:09 2026-02-26T03:11:18.149 INFO:teuthology.orchestra.run.trial053.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 20 kB 00:01 2026-02-26T03:11:19.398 INFO:teuthology.orchestra.run.trial053.stdout:Extra Packages for Enterprise Linux 17 MB/s | 20 MB 00:01 2026-02-26T03:11:22.603 INFO:teuthology.orchestra.run.trial053.stdout:lab-extras 953 kB/s | 50 kB 00:00 2026-02-26T03:11:23.794 INFO:teuthology.orchestra.run.trial053.stdout:Package ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:11:23.812 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:11:23.812 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:11:23.812 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:11:23.836 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install ceph-test 2026-02-26T03:11:24.154 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:00:02 ago on Thu 26 Feb 2026 03:11:22 AM UTC. 2026-02-26T03:11:24.219 INFO:teuthology.orchestra.run.trial053.stdout:Package ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:11:24.237 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:11:24.237 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:11:24.238 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:11:24.259 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install ceph 2026-02-26T03:11:24.581 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:00:02 ago on Thu 26 Feb 2026 03:11:22 AM UTC. 2026-02-26T03:11:24.646 INFO:teuthology.orchestra.run.trial053.stdout:Package ceph-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:11:24.664 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:11:24.664 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:11:24.664 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:11:24.686 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install ceph-base 2026-02-26T03:11:25.008 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:00:03 ago on Thu 26 Feb 2026 03:11:22 AM UTC. 2026-02-26T03:11:25.073 INFO:teuthology.orchestra.run.trial053.stdout:Package ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:11:25.091 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:11:25.092 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:11:25.092 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:11:25.112 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install cephadm 2026-02-26T03:11:25.426 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:00:03 ago on Thu 26 Feb 2026 03:11:22 AM UTC. 2026-02-26T03:11:25.491 INFO:teuthology.orchestra.run.trial053.stdout:Package cephadm-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:11:25.509 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:11:25.509 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:11:25.509 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:11:25.530 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install ceph-immutable-object-cache 2026-02-26T03:11:25.849 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:00:03 ago on Thu 26 Feb 2026 03:11:22 AM UTC. 2026-02-26T03:11:25.914 INFO:teuthology.orchestra.run.trial053.stdout:Package ceph-immutable-object-cache-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:11:25.932 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:11:25.932 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:11:25.933 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:11:25.954 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install ceph-mgr 2026-02-26T03:11:26.275 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:00:04 ago on Thu 26 Feb 2026 03:11:22 AM UTC. 2026-02-26T03:11:26.340 INFO:teuthology.orchestra.run.trial053.stdout:Package ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:11:26.357 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:11:26.358 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:11:26.358 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:11:26.379 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install ceph-mgr-dashboard 2026-02-26T03:11:26.693 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:00:04 ago on Thu 26 Feb 2026 03:11:22 AM UTC. 2026-02-26T03:11:26.758 INFO:teuthology.orchestra.run.trial053.stdout:Package ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:11:26.776 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:11:26.777 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:11:26.777 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:11:26.797 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-26T03:11:27.114 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:00:05 ago on Thu 26 Feb 2026 03:11:22 AM UTC. 2026-02-26T03:11:27.179 INFO:teuthology.orchestra.run.trial053.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:11:27.197 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:11:27.197 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:11:27.198 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:11:27.219 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install ceph-mgr-rook 2026-02-26T03:11:27.542 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:00:05 ago on Thu 26 Feb 2026 03:11:22 AM UTC. 2026-02-26T03:11:27.607 INFO:teuthology.orchestra.run.trial053.stdout:Package ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:11:27.625 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:11:27.625 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:11:27.625 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:11:27.647 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install ceph-mgr-cephadm 2026-02-26T03:11:27.964 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:00:05 ago on Thu 26 Feb 2026 03:11:22 AM UTC. 2026-02-26T03:11:28.029 INFO:teuthology.orchestra.run.trial053.stdout:Package ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:11:28.047 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:11:28.047 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:11:28.047 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:11:28.070 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install ceph-osd 2026-02-26T03:11:28.388 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:00:06 ago on Thu 26 Feb 2026 03:11:22 AM UTC. 2026-02-26T03:11:28.452 INFO:teuthology.orchestra.run.trial053.stdout:Package ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:11:28.470 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:11:28.471 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:11:28.471 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:11:28.493 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install ceph-osd-classic 2026-02-26T03:11:28.812 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:00:06 ago on Thu 26 Feb 2026 03:11:22 AM UTC. 2026-02-26T03:11:28.876 INFO:teuthology.orchestra.run.trial053.stdout:Package ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:11:28.894 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:11:28.895 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:11:28.895 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:11:28.918 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install ceph-fuse 2026-02-26T03:11:29.232 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:00:07 ago on Thu 26 Feb 2026 03:11:22 AM UTC. 2026-02-26T03:11:29.298 INFO:teuthology.orchestra.run.trial053.stdout:Package ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:11:29.315 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:11:29.316 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:11:29.316 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:11:29.338 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install ceph-volume 2026-02-26T03:11:29.658 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:00:07 ago on Thu 26 Feb 2026 03:11:22 AM UTC. 2026-02-26T03:11:29.724 INFO:teuthology.orchestra.run.trial053.stdout:Package ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:11:29.742 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:11:29.742 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:11:29.742 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:11:29.765 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install librados-devel 2026-02-26T03:11:30.081 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:00:08 ago on Thu 26 Feb 2026 03:11:22 AM UTC. 2026-02-26T03:11:30.147 INFO:teuthology.orchestra.run.trial053.stdout:Package librados-devel-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:11:30.165 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:11:30.166 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:11:30.166 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:11:30.190 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install libcephfs2 2026-02-26T03:11:30.509 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:00:08 ago on Thu 26 Feb 2026 03:11:22 AM UTC. 2026-02-26T03:11:30.576 INFO:teuthology.orchestra.run.trial053.stdout:Package libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:11:30.594 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:11:30.594 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:11:30.594 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:11:30.617 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install libcephfs-devel 2026-02-26T03:11:30.934 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:00:08 ago on Thu 26 Feb 2026 03:11:22 AM UTC. 2026-02-26T03:11:30.999 INFO:teuthology.orchestra.run.trial053.stdout:Package libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:11:31.017 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:11:31.017 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:11:31.017 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:11:31.039 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install librados2 2026-02-26T03:11:31.354 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:00:09 ago on Thu 26 Feb 2026 03:11:22 AM UTC. 2026-02-26T03:11:31.418 INFO:teuthology.orchestra.run.trial053.stdout:Package librados2-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:11:31.436 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:11:31.437 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:11:31.437 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:11:31.458 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install librbd1 2026-02-26T03:11:31.777 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:00:09 ago on Thu 26 Feb 2026 03:11:22 AM UTC. 2026-02-26T03:11:31.842 INFO:teuthology.orchestra.run.trial053.stdout:Package librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:11:31.860 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:11:31.860 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:11:31.860 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:11:31.881 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install python3-rados 2026-02-26T03:11:32.200 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:00:10 ago on Thu 26 Feb 2026 03:11:22 AM UTC. 2026-02-26T03:11:32.266 INFO:teuthology.orchestra.run.trial053.stdout:Package python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:11:32.283 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:11:32.284 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:11:32.285 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:11:32.306 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install python3-rgw 2026-02-26T03:11:32.625 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:00:10 ago on Thu 26 Feb 2026 03:11:22 AM UTC. 2026-02-26T03:11:32.691 INFO:teuthology.orchestra.run.trial053.stdout:Package python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:11:32.709 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:11:32.709 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:11:32.709 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:11:32.731 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install python3-cephfs 2026-02-26T03:11:33.047 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:00:11 ago on Thu 26 Feb 2026 03:11:22 AM UTC. 2026-02-26T03:11:33.112 INFO:teuthology.orchestra.run.trial053.stdout:Package python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:11:33.130 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:11:33.130 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:11:33.130 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:11:33.152 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install python3-rbd 2026-02-26T03:11:33.475 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:00:11 ago on Thu 26 Feb 2026 03:11:22 AM UTC. 2026-02-26T03:11:33.540 INFO:teuthology.orchestra.run.trial053.stdout:Package python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:11:33.558 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:11:33.558 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:11:33.559 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:11:33.581 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install rbd-fuse 2026-02-26T03:11:33.902 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:00:11 ago on Thu 26 Feb 2026 03:11:22 AM UTC. 2026-02-26T03:11:33.967 INFO:teuthology.orchestra.run.trial053.stdout:Package rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:11:33.985 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:11:33.985 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:11:33.985 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:11:34.007 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install rbd-mirror 2026-02-26T03:11:34.326 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:00:12 ago on Thu 26 Feb 2026 03:11:22 AM UTC. 2026-02-26T03:11:34.392 INFO:teuthology.orchestra.run.trial053.stdout:Package rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:11:34.410 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:11:34.410 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:11:34.410 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:11:34.432 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install rbd-nbd 2026-02-26T03:11:34.749 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:00:12 ago on Thu 26 Feb 2026 03:11:22 AM UTC. 2026-02-26T03:11:34.814 INFO:teuthology.orchestra.run.trial053.stdout:Package rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:11:34.832 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:11:34.832 INFO:teuthology.orchestra.run.trial053.stdout:Nothing to do. 2026-02-26T03:11:34.832 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:11:34.856 DEBUG:teuthology.parallel:result is None 2026-02-26T03:11:34.857 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:11:40.041 DEBUG:teuthology.orchestra.run.trial053:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-26T03:11:40.062 INFO:teuthology.orchestra.run.trial053.stdout:20.3.0-5312.gabffb617.el9 2026-02-26T03:11:40.063 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5312.gabffb617.el9 2026-02-26T03:11:40.063 INFO:teuthology.task.install:The correct ceph version 20.3.0-5312.gabffb617 is installed. 2026-02-26T03:11:40.066 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2026-02-26T03:11:40.066 INFO:teuthology.run_tasks:Running task ceph... 2026-02-26T03:11:40.177 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-26T03:11:40.177 DEBUG:teuthology.orchestra.run.trial053:> sudo chmod 777 /var/log/ceph 2026-02-26T03:11:40.203 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-26T03:11:40.203 DEBUG:teuthology.orchestra.run.trial053:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-26T03:11:40.266 INFO:tasks.ceph:Creating extra log directories... 2026-02-26T03:11:40.266 DEBUG:teuthology.orchestra.run.trial053:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-26T03:11:40.336 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-26T03:11:40.336 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-26T03:11:40.336 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71147', 'branch': 'main', 'description': 'crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm} perf_ignorelist settings/optimized workloads/fio_4K_rand_rw}', 'email': None, 'first_in_suite': False, 'flavor': 'debug', 'job_id': '71147', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996'}}, 'owner': 'scheduled_shraddhaag@soko04.front.sepia.ceph.com', 'priority': 101, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 5117, 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': '1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996', 'targets': {'trial053.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFp7yIyMFWu15PV8HBCA04CtZcJvdNOSgFjUcqQKEETY0bb8EkATuuC3ZGu4eapyulIJoMRRXAyt6P3JpRVzBlw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}}, {'pexec': {'all': ['sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson']}}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'wait-for-scrub': False, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8', 'cluster': 'ceph'}}, {'ssh_keys': None}, {'cbt': {'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randrw'], 'norandommap': True, 'op_size': [4096], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2]}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6e3d0419ccca0e75a03e08feb708946e198236f4', 'timestamp': '2026-02-26_02:22:56', 'tube': 'trial', 'user': 'shraddhaag', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699'} 2026-02-26T03:11:40.336 DEBUG:teuthology.orchestra.run.trial053:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-26T03:11:40.384 DEBUG:teuthology.orchestra.run.trial053:> sudo install -d -m0777 -- /var/run/ceph 2026-02-26T03:11:40.449 DEBUG:teuthology.orchestra.run.trial053:> set -ex 2026-02-26T03:11:40.449 DEBUG:teuthology.orchestra.run.trial053:> dd if=/scratch_devs of=/dev/stdout 2026-02-26T03:11:40.504 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-26T03:11:40.504 DEBUG:teuthology.orchestra.run.trial053:> stat /dev/vg_nvme/lv_1 2026-02-26T03:11:40.561 INFO:teuthology.orchestra.run.trial053.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-26T03:11:40.561 INFO:teuthology.orchestra.run.trial053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:11:40.561 INFO:teuthology.orchestra.run.trial053.stdout:Device: 6h/6d Inode: 1299 Links: 1 2026-02-26T03:11:40.561 INFO:teuthology.orchestra.run.trial053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:11:40.562 INFO:teuthology.orchestra.run.trial053.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:11:40.562 INFO:teuthology.orchestra.run.trial053.stdout:Access: 2026-02-26 03:07:52.966153155 +0000 2026-02-26T03:11:40.562 INFO:teuthology.orchestra.run.trial053.stdout:Modify: 2026-02-26 03:06:47.667386039 +0000 2026-02-26T03:11:40.562 INFO:teuthology.orchestra.run.trial053.stdout:Change: 2026-02-26 03:06:47.667386039 +0000 2026-02-26T03:11:40.562 INFO:teuthology.orchestra.run.trial053.stdout: Birth: 2026-02-26 03:06:47.667386039 +0000 2026-02-26T03:11:40.562 DEBUG:teuthology.orchestra.run.trial053:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-26T03:11:40.623 INFO:teuthology.orchestra.run.trial053.stderr:1+0 records in 2026-02-26T03:11:40.623 INFO:teuthology.orchestra.run.trial053.stderr:1+0 records out 2026-02-26T03:11:40.623 INFO:teuthology.orchestra.run.trial053.stderr:512 bytes copied, 0.000155078 s, 3.3 MB/s 2026-02-26T03:11:40.624 DEBUG:teuthology.orchestra.run.trial053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-26T03:11:40.681 DEBUG:teuthology.orchestra.run.trial053:> stat /dev/vg_nvme/lv_2 2026-02-26T03:11:40.738 INFO:teuthology.orchestra.run.trial053.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-26T03:11:40.738 INFO:teuthology.orchestra.run.trial053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:11:40.738 INFO:teuthology.orchestra.run.trial053.stdout:Device: 6h/6d Inode: 1308 Links: 1 2026-02-26T03:11:40.738 INFO:teuthology.orchestra.run.trial053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:11:40.738 INFO:teuthology.orchestra.run.trial053.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:11:40.738 INFO:teuthology.orchestra.run.trial053.stdout:Access: 2026-02-26 03:07:52.966153155 +0000 2026-02-26T03:11:40.738 INFO:teuthology.orchestra.run.trial053.stdout:Modify: 2026-02-26 03:06:47.668386000 +0000 2026-02-26T03:11:40.738 INFO:teuthology.orchestra.run.trial053.stdout:Change: 2026-02-26 03:06:47.668386000 +0000 2026-02-26T03:11:40.738 INFO:teuthology.orchestra.run.trial053.stdout: Birth: 2026-02-26 03:06:47.668386000 +0000 2026-02-26T03:11:40.738 DEBUG:teuthology.orchestra.run.trial053:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-26T03:11:40.799 INFO:teuthology.orchestra.run.trial053.stderr:1+0 records in 2026-02-26T03:11:40.800 INFO:teuthology.orchestra.run.trial053.stderr:1+0 records out 2026-02-26T03:11:40.800 INFO:teuthology.orchestra.run.trial053.stderr:512 bytes copied, 0.00016659 s, 3.1 MB/s 2026-02-26T03:11:40.801 DEBUG:teuthology.orchestra.run.trial053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-26T03:11:40.857 DEBUG:teuthology.orchestra.run.trial053:> stat /dev/vg_nvme/lv_3 2026-02-26T03:11:40.913 INFO:teuthology.orchestra.run.trial053.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-26T03:11:40.914 INFO:teuthology.orchestra.run.trial053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:11:40.914 INFO:teuthology.orchestra.run.trial053.stdout:Device: 6h/6d Inode: 1295 Links: 1 2026-02-26T03:11:40.914 INFO:teuthology.orchestra.run.trial053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:11:40.914 INFO:teuthology.orchestra.run.trial053.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:11:40.914 INFO:teuthology.orchestra.run.trial053.stdout:Access: 2026-02-26 03:07:52.967153117 +0000 2026-02-26T03:11:40.914 INFO:teuthology.orchestra.run.trial053.stdout:Modify: 2026-02-26 03:06:47.667386039 +0000 2026-02-26T03:11:40.914 INFO:teuthology.orchestra.run.trial053.stdout:Change: 2026-02-26 03:06:47.667386039 +0000 2026-02-26T03:11:40.914 INFO:teuthology.orchestra.run.trial053.stdout: Birth: 2026-02-26 03:06:47.667386039 +0000 2026-02-26T03:11:40.915 DEBUG:teuthology.orchestra.run.trial053:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-26T03:11:40.976 INFO:teuthology.orchestra.run.trial053.stderr:1+0 records in 2026-02-26T03:11:40.976 INFO:teuthology.orchestra.run.trial053.stderr:1+0 records out 2026-02-26T03:11:40.977 INFO:teuthology.orchestra.run.trial053.stderr:512 bytes copied, 0.000153124 s, 3.3 MB/s 2026-02-26T03:11:40.978 DEBUG:teuthology.orchestra.run.trial053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-26T03:11:41.034 DEBUG:teuthology.orchestra.run.trial053:> stat /dev/vg_nvme/lv_4 2026-02-26T03:11:41.090 INFO:teuthology.orchestra.run.trial053.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-26T03:11:41.090 INFO:teuthology.orchestra.run.trial053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:11:41.090 INFO:teuthology.orchestra.run.trial053.stdout:Device: 6h/6d Inode: 1309 Links: 1 2026-02-26T03:11:41.090 INFO:teuthology.orchestra.run.trial053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:11:41.090 INFO:teuthology.orchestra.run.trial053.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:11:41.090 INFO:teuthology.orchestra.run.trial053.stdout:Access: 2026-02-26 03:07:52.967153117 +0000 2026-02-26T03:11:41.090 INFO:teuthology.orchestra.run.trial053.stdout:Modify: 2026-02-26 03:06:47.668386000 +0000 2026-02-26T03:11:41.090 INFO:teuthology.orchestra.run.trial053.stdout:Change: 2026-02-26 03:06:47.668386000 +0000 2026-02-26T03:11:41.090 INFO:teuthology.orchestra.run.trial053.stdout: Birth: 2026-02-26 03:06:47.668386000 +0000 2026-02-26T03:11:41.091 DEBUG:teuthology.orchestra.run.trial053:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-26T03:11:41.153 INFO:teuthology.orchestra.run.trial053.stderr:1+0 records in 2026-02-26T03:11:41.153 INFO:teuthology.orchestra.run.trial053.stderr:1+0 records out 2026-02-26T03:11:41.153 INFO:teuthology.orchestra.run.trial053.stderr:512 bytes copied, 0.000147184 s, 3.5 MB/s 2026-02-26T03:11:41.154 DEBUG:teuthology.orchestra.run.trial053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-26T03:11:41.210 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:11:41.210 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial053.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-26T03:11:41.211 INFO:tasks.ceph:Generating config... 2026-02-26T03:11:41.211 INFO:tasks.ceph:[osd] debug monc = 20 2026-02-26T03:11:41.211 INFO:tasks.ceph:[osd] crimson cpu num = 4 2026-02-26T03:11:41.211 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2026-02-26T03:11:41.211 INFO:tasks.ceph:[osd] debug filestore = 0/0 2026-02-26T03:11:41.211 INFO:tasks.ceph:[osd] debug journal = 0/0 2026-02-26T03:11:41.211 INFO:tasks.ceph:[osd] debug ms = 0/0 2026-02-26T03:11:41.212 INFO:tasks.ceph:[osd] debug osd = 0/0 2026-02-26T03:11:41.212 INFO:tasks.ceph:[osd] debug seastore = 20 2026-02-26T03:11:41.212 INFO:tasks.ceph:[osd] debug seastore backref = 20 2026-02-26T03:11:41.212 INFO:tasks.ceph:[osd] debug seastore cache = 20 2026-02-26T03:11:41.212 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2026-02-26T03:11:41.212 INFO:tasks.ceph:[osd] debug seastore device = 20 2026-02-26T03:11:41.212 INFO:tasks.ceph:[osd] debug seastore epm = 20 2026-02-26T03:11:41.212 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2026-02-26T03:11:41.212 INFO:tasks.ceph:[osd] debug seastore journal = 20 2026-02-26T03:11:41.212 INFO:tasks.ceph:[osd] debug seastore lba = 20 2026-02-26T03:11:41.212 INFO:tasks.ceph:[osd] debug seastore odata = 20 2026-02-26T03:11:41.212 INFO:tasks.ceph:[osd] debug seastore omap = 20 2026-02-26T03:11:41.212 INFO:tasks.ceph:[osd] debug seastore onode = 20 2026-02-26T03:11:41.212 INFO:tasks.ceph:[osd] debug seastore t = 20 2026-02-26T03:11:41.212 INFO:tasks.ceph:[osd] debug seastore tm = 20 2026-02-26T03:11:41.212 INFO:tasks.ceph:[osd] osd objectstore = seastore 2026-02-26T03:11:41.212 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2026-02-26T03:11:41.212 INFO:tasks.ceph:[osd] seastore cachepin size pershard = 64M 2026-02-26T03:11:41.212 INFO:tasks.ceph:[osd] seastore main device type = RANDOM_BLOCK_SSD 2026-02-26T03:11:41.212 INFO:tasks.ceph:[osd] seastore max concurrent transactions = 8 2026-02-26T03:11:41.212 INFO:tasks.ceph:[global] auth client required = none 2026-02-26T03:11:41.212 INFO:tasks.ceph:[global] auth cluster required = none 2026-02-26T03:11:41.212 INFO:tasks.ceph:[global] auth service required = none 2026-02-26T03:11:41.212 INFO:tasks.ceph:[global] auth supported = none 2026-02-26T03:11:41.212 INFO:tasks.ceph:[global] debug asok = 0/0 2026-02-26T03:11:41.212 INFO:tasks.ceph:[global] debug auth = 0/0 2026-02-26T03:11:41.212 INFO:tasks.ceph:[global] debug bdev = 0/0 2026-02-26T03:11:41.212 INFO:tasks.ceph:[global] debug bluefs = 0/0 2026-02-26T03:11:41.212 INFO:tasks.ceph:[global] debug bluestore = 0/0 2026-02-26T03:11:41.212 INFO:tasks.ceph:[global] debug buffer = 0/0 2026-02-26T03:11:41.212 INFO:tasks.ceph:[global] debug civetweb = 0/0 2026-02-26T03:11:41.212 INFO:tasks.ceph:[global] debug client = 0/0 2026-02-26T03:11:41.212 INFO:tasks.ceph:[global] debug compressor = 0/0 2026-02-26T03:11:41.212 INFO:tasks.ceph:[global] debug context = 0/0 2026-02-26T03:11:41.212 INFO:tasks.ceph:[global] debug crush = 0/0 2026-02-26T03:11:41.212 INFO:tasks.ceph:[global] debug crypto = 0/0 2026-02-26T03:11:41.212 INFO:tasks.ceph:[global] debug dpdk = 0/0 2026-02-26T03:11:41.212 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2026-02-26T03:11:41.212 INFO:tasks.ceph:[global] debug filer = 0/0 2026-02-26T03:11:41.212 INFO:tasks.ceph:[global] debug filestore = 0/0 2026-02-26T03:11:41.212 INFO:tasks.ceph:[global] debug finisher = 0/0 2026-02-26T03:11:41.212 INFO:tasks.ceph:[global] debug fuse = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug javaclient = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug journal = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug journaler = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug kstore = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug leveldb = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug lockdep = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug mds = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug mds locker = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug mds log = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug memdb = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug mgr = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug mgrc = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug mon = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug monc = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug ms = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug objclass = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug objecter = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug optracker = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug osd = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug paxos = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug rados = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug rbd = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug refs = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug rgw = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug striper = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug throttle = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug timer = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] debug tp = 0/0 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] ms client mode = crc 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] ms cluster mode = crc 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] ms mon client mode = crc 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2026-02-26T03:11:41.213 INFO:tasks.ceph:[global] ms mon service mode = crc 2026-02-26T03:11:41.214 INFO:tasks.ceph:[global] ms service mode = crc 2026-02-26T03:11:41.214 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-26T03:11:41.214 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-26T03:11:41.214 INFO:tasks.ceph:[mon] debug mon = 0/0 2026-02-26T03:11:41.214 INFO:tasks.ceph:[mon] debug ms = 0/0 2026-02-26T03:11:41.214 INFO:tasks.ceph:[mon] debug paxos = 0/0 2026-02-26T03:11:41.214 INFO:tasks.ceph:[mon] osd pool default crimson = True 2026-02-26T03:11:41.214 INFO:tasks.ceph:Setting up mon.a... 2026-02-26T03:11:41.214 DEBUG:teuthology.orchestra.run.trial053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-26T03:11:41.292 INFO:teuthology.orchestra.run.trial053.stdout:creating /etc/ceph/ceph.keyring 2026-02-26T03:11:41.295 DEBUG:teuthology.orchestra.run.trial053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-26T03:11:41.378 DEBUG:teuthology.orchestra.run.trial053:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-26T03:11:41.442 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.53')] 2026-02-26T03:11:41.442 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.53', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-26T03:11:41.443 DEBUG:teuthology.orchestra.run.trial053:> set -ex 2026-02-26T03:11:41.443 DEBUG:teuthology.orchestra.run.trial053:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-26T03:11:41.497 DEBUG:teuthology.orchestra.run.trial053:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.53 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:11:41.578 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:11:41.576+0000 7fd856e2af40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:11:41.578 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:11:41.576+0000 7fd856e2af40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:11:41.578 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:11:41.576+0000 7fd856e2af40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:11:41.579 INFO:teuthology.orchestra.run.trial053.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:11:41.579 INFO:teuthology.orchestra.run.trial053.stdout:monmaptool: generated fsid 10bcb95a-6ed5-4257-8ca2-306c33baed91 2026-02-26T03:11:41.579 INFO:teuthology.orchestra.run.trial053.stdout:setting min_mon_release = tentacle 2026-02-26T03:11:41.579 INFO:teuthology.orchestra.run.trial053.stdout:epoch 0 2026-02-26T03:11:41.579 INFO:teuthology.orchestra.run.trial053.stdout:fsid 10bcb95a-6ed5-4257-8ca2-306c33baed91 2026-02-26T03:11:41.579 INFO:teuthology.orchestra.run.trial053.stdout:last_changed 2026-02-26T03:11:41.578150+0000 2026-02-26T03:11:41.579 INFO:teuthology.orchestra.run.trial053.stdout:created 2026-02-26T03:11:41.578150+0000 2026-02-26T03:11:41.579 INFO:teuthology.orchestra.run.trial053.stdout:min_mon_release 20 (tentacle) 2026-02-26T03:11:41.579 INFO:teuthology.orchestra.run.trial053.stdout:election_strategy: 1 2026-02-26T03:11:41.579 INFO:teuthology.orchestra.run.trial053.stdout:0: [v2:10.20.193.53:3300/0,v1:10.20.193.53:6789/0] mon.a 2026-02-26T03:11:41.579 INFO:teuthology.orchestra.run.trial053.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-26T03:11:41.580 DEBUG:teuthology.orchestra.run.trial053:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-26T03:11:41.635 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 10bcb95a-6ed5-4257-8ca2-306c33baed91... 2026-02-26T03:11:41.636 DEBUG:teuthology.orchestra.run.trial053:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-26T03:11:41.717 INFO:teuthology.orchestra.run.trial053.stdout:[global] 2026-02-26T03:11:41.717 INFO:teuthology.orchestra.run.trial053.stdout: chdir = "" 2026-02-26T03:11:41.717 INFO:teuthology.orchestra.run.trial053.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-26T03:11:41.717 INFO:teuthology.orchestra.run.trial053.stdout: auth supported = none 2026-02-26T03:11:41.718 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:11:41.718 INFO:teuthology.orchestra.run.trial053.stdout: filestore xattr use omap = true 2026-02-26T03:11:41.718 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:11:41.718 INFO:teuthology.orchestra.run.trial053.stdout: mon clock drift allowed = 1.000 2026-02-26T03:11:41.718 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:11:41.718 INFO:teuthology.orchestra.run.trial053.stdout: osd crush chooseleaf type = 0 2026-02-26T03:11:41.718 INFO:teuthology.orchestra.run.trial053.stdout: auth debug = true 2026-02-26T03:11:41.718 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:11:41.718 INFO:teuthology.orchestra.run.trial053.stdout: ms die on old message = true 2026-02-26T03:11:41.718 INFO:teuthology.orchestra.run.trial053.stdout: ms die on bug = true 2026-02-26T03:11:41.718 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:11:41.718 INFO:teuthology.orchestra.run.trial053.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-26T03:11:41.718 INFO:teuthology.orchestra.run.trial053.stdout: mon pg warn max object skew = 0 2026-02-26T03:11:41.718 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:11:41.718 INFO:teuthology.orchestra.run.trial053.stdout: # disable pg_autoscaler by default for new pools 2026-02-26T03:11:41.718 INFO:teuthology.orchestra.run.trial053.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-26T03:11:41.718 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:11:41.718 INFO:teuthology.orchestra.run.trial053.stdout: osd pool default size = 2 2026-02-26T03:11:41.718 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:11:41.719 INFO:teuthology.orchestra.run.trial053.stdout: mon osd allow primary affinity = true 2026-02-26T03:11:41.719 INFO:teuthology.orchestra.run.trial053.stdout: mon osd allow pg remap = true 2026-02-26T03:11:41.719 INFO:teuthology.orchestra.run.trial053.stdout: mon warn on legacy crush tunables = false 2026-02-26T03:11:41.719 INFO:teuthology.orchestra.run.trial053.stdout: mon warn on crush straw calc version zero = false 2026-02-26T03:11:41.719 INFO:teuthology.orchestra.run.trial053.stdout: mon warn on no sortbitwise = false 2026-02-26T03:11:41.719 INFO:teuthology.orchestra.run.trial053.stdout: mon warn on osd down out interval zero = false 2026-02-26T03:11:41.719 INFO:teuthology.orchestra.run.trial053.stdout: mon warn on too few osds = false 2026-02-26T03:11:41.719 INFO:teuthology.orchestra.run.trial053.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-26T03:11:41.719 INFO:teuthology.orchestra.run.trial053.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-26T03:11:41.719 INFO:teuthology.orchestra.run.trial053.stdout: mon_allow_pool_size_one = true 2026-02-26T03:11:41.719 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:11:41.719 INFO:teuthology.orchestra.run.trial053.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-26T03:11:41.719 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:11:41.719 INFO:teuthology.orchestra.run.trial053.stdout: osd default data pool replay window = 5 2026-02-26T03:11:41.719 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:11:41.719 INFO:teuthology.orchestra.run.trial053.stdout: mon allow pool delete = true 2026-02-26T03:11:41.719 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:11:41.719 INFO:teuthology.orchestra.run.trial053.stdout: mon cluster log file level = debug 2026-02-26T03:11:41.719 INFO:teuthology.orchestra.run.trial053.stdout: debug asserts on shutdown = true 2026-02-26T03:11:41.719 INFO:teuthology.orchestra.run.trial053.stdout: mon health detail to clog = false 2026-02-26T03:11:41.719 INFO:teuthology.orchestra.run.trial053.stdout: mon host = 10.20.193.53 2026-02-26T03:11:41.719 INFO:teuthology.orchestra.run.trial053.stdout: auth client required = none 2026-02-26T03:11:41.719 INFO:teuthology.orchestra.run.trial053.stdout: auth cluster required = none 2026-02-26T03:11:41.720 INFO:teuthology.orchestra.run.trial053.stdout: auth service required = none 2026-02-26T03:11:41.720 INFO:teuthology.orchestra.run.trial053.stdout: debug asok = 0/0 2026-02-26T03:11:41.720 INFO:teuthology.orchestra.run.trial053.stdout: debug auth = 0/0 2026-02-26T03:11:41.720 INFO:teuthology.orchestra.run.trial053.stdout: debug bdev = 0/0 2026-02-26T03:11:41.720 INFO:teuthology.orchestra.run.trial053.stdout: debug bluefs = 0/0 2026-02-26T03:11:41.720 INFO:teuthology.orchestra.run.trial053.stdout: debug bluestore = 0/0 2026-02-26T03:11:41.720 INFO:teuthology.orchestra.run.trial053.stdout: debug buffer = 0/0 2026-02-26T03:11:41.720 INFO:teuthology.orchestra.run.trial053.stdout: debug civetweb = 0/0 2026-02-26T03:11:41.720 INFO:teuthology.orchestra.run.trial053.stdout: debug client = 0/0 2026-02-26T03:11:41.720 INFO:teuthology.orchestra.run.trial053.stdout: debug compressor = 0/0 2026-02-26T03:11:41.720 INFO:teuthology.orchestra.run.trial053.stdout: debug context = 0/0 2026-02-26T03:11:41.720 INFO:teuthology.orchestra.run.trial053.stdout: debug crush = 0/0 2026-02-26T03:11:41.720 INFO:teuthology.orchestra.run.trial053.stdout: debug crypto = 0/0 2026-02-26T03:11:41.720 INFO:teuthology.orchestra.run.trial053.stdout: debug dpdk = 0/0 2026-02-26T03:11:41.720 INFO:teuthology.orchestra.run.trial053.stdout: debug eventtrace = 0/0 2026-02-26T03:11:41.720 INFO:teuthology.orchestra.run.trial053.stdout: debug filer = 0/0 2026-02-26T03:11:41.720 INFO:teuthology.orchestra.run.trial053.stdout: debug filestore = 0/0 2026-02-26T03:11:41.720 INFO:teuthology.orchestra.run.trial053.stdout: debug finisher = 0/0 2026-02-26T03:11:41.720 INFO:teuthology.orchestra.run.trial053.stdout: debug fuse = 0/0 2026-02-26T03:11:41.720 INFO:teuthology.orchestra.run.trial053.stdout: debug heartbeatmap = 0/0 2026-02-26T03:11:41.720 INFO:teuthology.orchestra.run.trial053.stdout: debug javaclient = 0/0 2026-02-26T03:11:41.720 INFO:teuthology.orchestra.run.trial053.stdout: debug journal = 0/0 2026-02-26T03:11:41.720 INFO:teuthology.orchestra.run.trial053.stdout: debug journaler = 0/0 2026-02-26T03:11:41.720 INFO:teuthology.orchestra.run.trial053.stdout: debug kstore = 0/0 2026-02-26T03:11:41.721 INFO:teuthology.orchestra.run.trial053.stdout: debug leveldb = 0/0 2026-02-26T03:11:41.721 INFO:teuthology.orchestra.run.trial053.stdout: debug lockdep = 0/0 2026-02-26T03:11:41.721 INFO:teuthology.orchestra.run.trial053.stdout: debug mds = 0/0 2026-02-26T03:11:41.721 INFO:teuthology.orchestra.run.trial053.stdout: debug mds balancer = 0/0 2026-02-26T03:11:41.721 INFO:teuthology.orchestra.run.trial053.stdout: debug mds locker = 0/0 2026-02-26T03:11:41.721 INFO:teuthology.orchestra.run.trial053.stdout: debug mds log = 0/0 2026-02-26T03:11:41.721 INFO:teuthology.orchestra.run.trial053.stdout: debug mds log expire = 0/0 2026-02-26T03:11:41.721 INFO:teuthology.orchestra.run.trial053.stdout: debug mds migrator = 0/0 2026-02-26T03:11:41.721 INFO:teuthology.orchestra.run.trial053.stdout: debug memdb = 0/0 2026-02-26T03:11:41.721 INFO:teuthology.orchestra.run.trial053.stdout: debug mgr = 0/0 2026-02-26T03:11:41.721 INFO:teuthology.orchestra.run.trial053.stdout: debug mgrc = 0/0 2026-02-26T03:11:41.721 INFO:teuthology.orchestra.run.trial053.stdout: debug mon = 0/0 2026-02-26T03:11:41.721 INFO:teuthology.orchestra.run.trial053.stdout: debug monc = 0/0 2026-02-26T03:11:41.721 INFO:teuthology.orchestra.run.trial053.stdout: debug ms = 0/0 2026-02-26T03:11:41.721 INFO:teuthology.orchestra.run.trial053.stdout: debug objclass = 0/0 2026-02-26T03:11:41.721 INFO:teuthology.orchestra.run.trial053.stdout: debug objectcacher = 0/0 2026-02-26T03:11:41.721 INFO:teuthology.orchestra.run.trial053.stdout: debug objecter = 0/0 2026-02-26T03:11:41.721 INFO:teuthology.orchestra.run.trial053.stdout: debug optracker = 0/0 2026-02-26T03:11:41.721 INFO:teuthology.orchestra.run.trial053.stdout: debug osd = 0/0 2026-02-26T03:11:41.721 INFO:teuthology.orchestra.run.trial053.stdout: debug paxos = 0/0 2026-02-26T03:11:41.721 INFO:teuthology.orchestra.run.trial053.stdout: debug perfcounter = 0/0 2026-02-26T03:11:41.721 INFO:teuthology.orchestra.run.trial053.stdout: debug rados = 0/0 2026-02-26T03:11:41.721 INFO:teuthology.orchestra.run.trial053.stdout: debug rbd = 0/0 2026-02-26T03:11:41.721 INFO:teuthology.orchestra.run.trial053.stdout: debug rbd mirror = 0/0 2026-02-26T03:11:41.722 INFO:teuthology.orchestra.run.trial053.stdout: debug rbd replay = 0/0 2026-02-26T03:11:41.722 INFO:teuthology.orchestra.run.trial053.stdout: debug refs = 0/0 2026-02-26T03:11:41.722 INFO:teuthology.orchestra.run.trial053.stdout: debug rgw = 0/0 2026-02-26T03:11:41.722 INFO:teuthology.orchestra.run.trial053.stdout: debug rgw lifecycle = 0/0 2026-02-26T03:11:41.722 INFO:teuthology.orchestra.run.trial053.stdout: debug rgw notification = 0/0 2026-02-26T03:11:41.722 INFO:teuthology.orchestra.run.trial053.stdout: debug rgw sync = 0/0 2026-02-26T03:11:41.722 INFO:teuthology.orchestra.run.trial053.stdout: debug rocksdb = 0/0 2026-02-26T03:11:41.722 INFO:teuthology.orchestra.run.trial053.stdout: debug striper = 0/0 2026-02-26T03:11:41.722 INFO:teuthology.orchestra.run.trial053.stdout: debug throttle = 0/0 2026-02-26T03:11:41.722 INFO:teuthology.orchestra.run.trial053.stdout: debug timer = 0/0 2026-02-26T03:11:41.722 INFO:teuthology.orchestra.run.trial053.stdout: debug tp = 0/0 2026-02-26T03:11:41.722 INFO:teuthology.orchestra.run.trial053.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-26T03:11:41.722 INFO:teuthology.orchestra.run.trial053.stdout: ms client mode = crc 2026-02-26T03:11:41.722 INFO:teuthology.orchestra.run.trial053.stdout: ms cluster mode = crc 2026-02-26T03:11:41.722 INFO:teuthology.orchestra.run.trial053.stdout: ms mon client mode = crc 2026-02-26T03:11:41.722 INFO:teuthology.orchestra.run.trial053.stdout: ms mon cluster mode = crc 2026-02-26T03:11:41.722 INFO:teuthology.orchestra.run.trial053.stdout: ms mon service mode = crc 2026-02-26T03:11:41.722 INFO:teuthology.orchestra.run.trial053.stdout: ms service mode = crc 2026-02-26T03:11:41.722 INFO:teuthology.orchestra.run.trial053.stdout: fsid = 10bcb95a-6ed5-4257-8ca2-306c33baed91 2026-02-26T03:11:41.722 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:11:41.722 INFO:teuthology.orchestra.run.trial053.stdout:[osd] 2026-02-26T03:11:41.722 INFO:teuthology.orchestra.run.trial053.stdout: osd journal size = 100 2026-02-26T03:11:41.722 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:11:41.722 INFO:teuthology.orchestra.run.trial053.stdout: osd scrub load threshold = 5.0 2026-02-26T03:11:41.723 INFO:teuthology.orchestra.run.trial053.stdout: osd scrub max interval = 600 2026-02-26T03:11:41.723 INFO:teuthology.orchestra.run.trial053.stdout: osd mclock profile = high_recovery_ops 2026-02-26T03:11:41.723 INFO:teuthology.orchestra.run.trial053.stdout: osd mclock skip benchmark = true 2026-02-26T03:11:41.723 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:11:41.723 INFO:teuthology.orchestra.run.trial053.stdout: osd recover clone overlap = true 2026-02-26T03:11:41.723 INFO:teuthology.orchestra.run.trial053.stdout: osd recovery max chunk = 1048576 2026-02-26T03:11:41.723 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:11:41.723 INFO:teuthology.orchestra.run.trial053.stdout: osd debug shutdown = true 2026-02-26T03:11:41.723 INFO:teuthology.orchestra.run.trial053.stdout: osd debug op order = true 2026-02-26T03:11:41.723 INFO:teuthology.orchestra.run.trial053.stdout: osd debug verify stray on activate = true 2026-02-26T03:11:41.723 INFO:teuthology.orchestra.run.trial053.stdout: osd debug trim objects = true 2026-02-26T03:11:41.723 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:11:41.723 INFO:teuthology.orchestra.run.trial053.stdout: osd open classes on start = true 2026-02-26T03:11:41.723 INFO:teuthology.orchestra.run.trial053.stdout: osd debug pg log writeout = true 2026-02-26T03:11:41.723 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:11:41.723 INFO:teuthology.orchestra.run.trial053.stdout: osd deep scrub update digest min age = 30 2026-02-26T03:11:41.723 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:11:41.723 INFO:teuthology.orchestra.run.trial053.stdout: osd map max advance = 10 2026-02-26T03:11:41.723 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:11:41.723 INFO:teuthology.orchestra.run.trial053.stdout: journal zero on create = true 2026-02-26T03:11:41.723 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:11:41.723 INFO:teuthology.orchestra.run.trial053.stdout: filestore ondisk finisher threads = 3 2026-02-26T03:11:41.723 INFO:teuthology.orchestra.run.trial053.stdout: filestore apply finisher threads = 3 2026-02-26T03:11:41.723 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:11:41.724 INFO:teuthology.orchestra.run.trial053.stdout: bdev debug aio = true 2026-02-26T03:11:41.724 INFO:teuthology.orchestra.run.trial053.stdout: osd debug misdirected ops = true 2026-02-26T03:11:41.724 INFO:teuthology.orchestra.run.trial053.stdout: debug monc = 20 2026-02-26T03:11:41.724 INFO:teuthology.orchestra.run.trial053.stdout: crimson cpu num = 4 2026-02-26T03:11:41.724 INFO:teuthology.orchestra.run.trial053.stdout: crimson osd obc lru size = 10 2026-02-26T03:11:41.724 INFO:teuthology.orchestra.run.trial053.stdout: debug filestore = 0/0 2026-02-26T03:11:41.724 INFO:teuthology.orchestra.run.trial053.stdout: debug journal = 0/0 2026-02-26T03:11:41.724 INFO:teuthology.orchestra.run.trial053.stdout: debug ms = 0/0 2026-02-26T03:11:41.724 INFO:teuthology.orchestra.run.trial053.stdout: debug osd = 0/0 2026-02-26T03:11:41.724 INFO:teuthology.orchestra.run.trial053.stdout: debug seastore = 20 2026-02-26T03:11:41.724 INFO:teuthology.orchestra.run.trial053.stdout: debug seastore backref = 20 2026-02-26T03:11:41.724 INFO:teuthology.orchestra.run.trial053.stdout: debug seastore cache = 20 2026-02-26T03:11:41.724 INFO:teuthology.orchestra.run.trial053.stdout: debug seastore cleaner = 20 2026-02-26T03:11:41.724 INFO:teuthology.orchestra.run.trial053.stdout: debug seastore device = 20 2026-02-26T03:11:41.724 INFO:teuthology.orchestra.run.trial053.stdout: debug seastore epm = 20 2026-02-26T03:11:41.724 INFO:teuthology.orchestra.run.trial053.stdout: debug seastore fixedkv tree = 20 2026-02-26T03:11:41.724 INFO:teuthology.orchestra.run.trial053.stdout: debug seastore journal = 20 2026-02-26T03:11:41.724 INFO:teuthology.orchestra.run.trial053.stdout: debug seastore lba = 20 2026-02-26T03:11:41.724 INFO:teuthology.orchestra.run.trial053.stdout: debug seastore odata = 20 2026-02-26T03:11:41.724 INFO:teuthology.orchestra.run.trial053.stdout: debug seastore omap = 20 2026-02-26T03:11:41.724 INFO:teuthology.orchestra.run.trial053.stdout: debug seastore onode = 20 2026-02-26T03:11:41.724 INFO:teuthology.orchestra.run.trial053.stdout: debug seastore t = 20 2026-02-26T03:11:41.724 INFO:teuthology.orchestra.run.trial053.stdout: debug seastore tm = 20 2026-02-26T03:11:41.724 INFO:teuthology.orchestra.run.trial053.stdout: osd objectstore = seastore 2026-02-26T03:11:41.725 INFO:teuthology.orchestra.run.trial053.stdout: osd shutdown pgref assert = True 2026-02-26T03:11:41.725 INFO:teuthology.orchestra.run.trial053.stdout: seastore cachepin size pershard = 64M 2026-02-26T03:11:41.725 INFO:teuthology.orchestra.run.trial053.stdout: seastore main device type = RANDOM_BLOCK_SSD 2026-02-26T03:11:41.725 INFO:teuthology.orchestra.run.trial053.stdout: seastore max concurrent transactions = 8 2026-02-26T03:11:41.725 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:11:41.725 INFO:teuthology.orchestra.run.trial053.stdout:[mgr] 2026-02-26T03:11:41.725 INFO:teuthology.orchestra.run.trial053.stdout: debug ms = 1 2026-02-26T03:11:41.725 INFO:teuthology.orchestra.run.trial053.stdout: debug mgr = 20 2026-02-26T03:11:41.725 INFO:teuthology.orchestra.run.trial053.stdout: debug mon = 20 2026-02-26T03:11:41.725 INFO:teuthology.orchestra.run.trial053.stdout: debug auth = 20 2026-02-26T03:11:41.725 INFO:teuthology.orchestra.run.trial053.stdout: mon reweight min pgs per osd = 4 2026-02-26T03:11:41.725 INFO:teuthology.orchestra.run.trial053.stdout: mon reweight min bytes per osd = 10 2026-02-26T03:11:41.725 INFO:teuthology.orchestra.run.trial053.stdout: mgr/telemetry/nag = false 2026-02-26T03:11:41.725 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:11:41.725 INFO:teuthology.orchestra.run.trial053.stdout:[mon] 2026-02-26T03:11:41.725 INFO:teuthology.orchestra.run.trial053.stdout: debug ms = 0/0 2026-02-26T03:11:41.725 INFO:teuthology.orchestra.run.trial053.stdout: debug mon = 0/0 2026-02-26T03:11:41.725 INFO:teuthology.orchestra.run.trial053.stdout: debug paxos = 0/0 2026-02-26T03:11:41.725 INFO:teuthology.orchestra.run.trial053.stdout: debug auth = 20 2026-02-26T03:11:41.725 INFO:teuthology.orchestra.run.trial053.stdout: mon data avail warn = 5 2026-02-26T03:11:41.725 INFO:teuthology.orchestra.run.trial053.stdout: mon mgr mkfs grace = 240 2026-02-26T03:11:41.725 INFO:teuthology.orchestra.run.trial053.stdout: mon reweight min pgs per osd = 4 2026-02-26T03:11:41.725 INFO:teuthology.orchestra.run.trial053.stdout: mon osd reporter subtree level = osd 2026-02-26T03:11:41.726 INFO:teuthology.orchestra.run.trial053.stdout: mon osd prime pg temp = true 2026-02-26T03:11:41.726 INFO:teuthology.orchestra.run.trial053.stdout: mon reweight min bytes per osd = 10 2026-02-26T03:11:41.726 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:11:41.726 INFO:teuthology.orchestra.run.trial053.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-26T03:11:41.726 INFO:teuthology.orchestra.run.trial053.stdout: auth mon ticket ttl = 660 # 11m 2026-02-26T03:11:41.726 INFO:teuthology.orchestra.run.trial053.stdout: auth service ticket ttl = 240 # 4m 2026-02-26T03:11:41.726 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:11:41.726 INFO:teuthology.orchestra.run.trial053.stdout: # don't complain about insecure global_id in the test suite 2026-02-26T03:11:41.726 INFO:teuthology.orchestra.run.trial053.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-26T03:11:41.726 INFO:teuthology.orchestra.run.trial053.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-26T03:11:41.726 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:11:41.726 INFO:teuthology.orchestra.run.trial053.stdout: # 1m isn't quite enough 2026-02-26T03:11:41.726 INFO:teuthology.orchestra.run.trial053.stdout: mon_down_mkfs_grace = 2m 2026-02-26T03:11:41.726 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:11:41.726 INFO:teuthology.orchestra.run.trial053.stdout: mon_warn_on_filestore_osds = false 2026-02-26T03:11:41.726 INFO:teuthology.orchestra.run.trial053.stdout: osd pool default crimson = True 2026-02-26T03:11:41.726 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:11:41.726 INFO:teuthology.orchestra.run.trial053.stdout:[client] 2026-02-26T03:11:41.726 INFO:teuthology.orchestra.run.trial053.stdout: rgw cache enabled = true 2026-02-26T03:11:41.726 INFO:teuthology.orchestra.run.trial053.stdout: rgw enable ops log = true 2026-02-26T03:11:41.726 INFO:teuthology.orchestra.run.trial053.stdout: rgw enable usage log = true 2026-02-26T03:11:41.726 INFO:teuthology.orchestra.run.trial053.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-26T03:11:41.726 INFO:teuthology.orchestra.run.trial053.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-26T03:11:41.726 INFO:teuthology.orchestra.run.trial053.stdout:[mon.a] 2026-02-26T03:11:41.727 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-26T03:11:41.727 DEBUG:teuthology.orchestra.run.trial053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-26T03:11:41.819 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-26T03:11:41.820 DEBUG:teuthology.orchestra.run.trial053:> set -ex 2026-02-26T03:11:41.820 DEBUG:teuthology.orchestra.run.trial053:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-26T03:11:41.836 DEBUG:teuthology.orchestra.run.trial053:> set -ex 2026-02-26T03:11:41.836 DEBUG:teuthology.orchestra.run.trial053:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-26T03:11:41.892 INFO:tasks.ceph:Sending monmap to node ubuntu@trial053.front.sepia.ceph.com 2026-02-26T03:11:41.892 DEBUG:teuthology.orchestra.run.trial053:> set -ex 2026-02-26T03:11:41.892 DEBUG:teuthology.orchestra.run.trial053:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-26T03:11:41.892 DEBUG:teuthology.orchestra.run.trial053:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-26T03:11:41.966 DEBUG:teuthology.orchestra.run.trial053:> set -ex 2026-02-26T03:11:41.967 DEBUG:teuthology.orchestra.run.trial053:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:11:42.022 INFO:tasks.ceph:Setting up mon nodes... 2026-02-26T03:11:42.022 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-26T03:11:42.023 DEBUG:teuthology.orchestra.run.trial053:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-26T03:11:42.113 INFO:teuthology.orchestra.run.trial053.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-26T03:11:42.116 INFO:tasks.ceph:Setting up mds nodes... 2026-02-26T03:11:42.116 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-26T03:11:42.116 DEBUG:teuthology.orchestra.run.trial053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-26T03:11:42.158 INFO:teuthology.orchestra.run.trial053.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-26T03:11:42.170 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-26T03:11:42.170 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial053.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-26T03:11:42.170 DEBUG:teuthology.orchestra.run.trial053:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-26T03:11:42.235 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:11:42.235 INFO:tasks.ceph:role: osd.0 2026-02-26T03:11:42.235 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial053.front.sepia.ceph.com 2026-02-26T03:11:42.235 DEBUG:teuthology.orchestra.run.trial053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-26T03:11:42.301 INFO:teuthology.orchestra.run.trial053.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-26T03:11:42.301 INFO:teuthology.orchestra.run.trial053.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-26T03:11:42.301 INFO:teuthology.orchestra.run.trial053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-26T03:11:42.301 INFO:teuthology.orchestra.run.trial053.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-26T03:11:42.301 INFO:teuthology.orchestra.run.trial053.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-26T03:11:42.301 INFO:teuthology.orchestra.run.trial053.stdout: = sunit=0 swidth=0 blks 2026-02-26T03:11:42.302 INFO:teuthology.orchestra.run.trial053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-26T03:11:42.302 INFO:teuthology.orchestra.run.trial053.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-26T03:11:42.302 INFO:teuthology.orchestra.run.trial053.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-26T03:11:42.302 INFO:teuthology.orchestra.run.trial053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-26T03:11:42.918 INFO:teuthology.orchestra.run.trial053.stdout:Discarding blocks...Done. 2026-02-26T03:11:42.920 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial053.front.sepia.ceph.com -o noatime 2026-02-26T03:11:42.920 DEBUG:teuthology.orchestra.run.trial053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-26T03:11:42.988 DEBUG:teuthology.orchestra.run.trial053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-26T03:11:43.055 DEBUG:teuthology.orchestra.run.trial053:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-26T03:11:43.121 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:11:43.121 INFO:tasks.ceph:role: osd.1 2026-02-26T03:11:43.121 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial053.front.sepia.ceph.com 2026-02-26T03:11:43.121 DEBUG:teuthology.orchestra.run.trial053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-26T03:11:43.187 INFO:teuthology.orchestra.run.trial053.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-26T03:11:43.187 INFO:teuthology.orchestra.run.trial053.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-26T03:11:43.187 INFO:teuthology.orchestra.run.trial053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-26T03:11:43.187 INFO:teuthology.orchestra.run.trial053.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-26T03:11:43.188 INFO:teuthology.orchestra.run.trial053.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-26T03:11:43.188 INFO:teuthology.orchestra.run.trial053.stdout: = sunit=0 swidth=0 blks 2026-02-26T03:11:43.188 INFO:teuthology.orchestra.run.trial053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-26T03:11:43.188 INFO:teuthology.orchestra.run.trial053.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-26T03:11:43.188 INFO:teuthology.orchestra.run.trial053.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-26T03:11:43.188 INFO:teuthology.orchestra.run.trial053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-26T03:11:43.805 INFO:teuthology.orchestra.run.trial053.stdout:Discarding blocks...Done. 2026-02-26T03:11:43.807 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial053.front.sepia.ceph.com -o noatime 2026-02-26T03:11:43.807 DEBUG:teuthology.orchestra.run.trial053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-26T03:11:43.876 DEBUG:teuthology.orchestra.run.trial053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-26T03:11:43.943 DEBUG:teuthology.orchestra.run.trial053:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-26T03:11:44.006 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:11:44.006 INFO:tasks.ceph:role: osd.2 2026-02-26T03:11:44.006 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial053.front.sepia.ceph.com 2026-02-26T03:11:44.006 DEBUG:teuthology.orchestra.run.trial053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-26T03:11:44.071 INFO:teuthology.orchestra.run.trial053.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-26T03:11:44.071 INFO:teuthology.orchestra.run.trial053.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-26T03:11:44.071 INFO:teuthology.orchestra.run.trial053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-26T03:11:44.071 INFO:teuthology.orchestra.run.trial053.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-26T03:11:44.071 INFO:teuthology.orchestra.run.trial053.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-26T03:11:44.071 INFO:teuthology.orchestra.run.trial053.stdout: = sunit=0 swidth=0 blks 2026-02-26T03:11:44.071 INFO:teuthology.orchestra.run.trial053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-26T03:11:44.071 INFO:teuthology.orchestra.run.trial053.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-26T03:11:44.071 INFO:teuthology.orchestra.run.trial053.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-26T03:11:44.071 INFO:teuthology.orchestra.run.trial053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-26T03:11:44.631 INFO:teuthology.orchestra.run.trial053.stdout:Discarding blocks...Done. 2026-02-26T03:11:44.633 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial053.front.sepia.ceph.com -o noatime 2026-02-26T03:11:44.633 DEBUG:teuthology.orchestra.run.trial053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-26T03:11:44.699 DEBUG:teuthology.orchestra.run.trial053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-26T03:11:44.766 DEBUG:teuthology.orchestra.run.trial053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:11:44.915 INFO:teuthology.orchestra.run.trial053.stderr:INFO  osd - OSD::main: parsing early config 2026-02-26T03:11:44.968 INFO:teuthology.orchestra.run.trial053.stderr:INFO 2026-02-26 03:11:44,967 seastar - Reactor backend: linux-aio 2026-02-26T03:11:44.985 INFO:teuthology.orchestra.run.trial053.stderr:INFO 2026-02-26 03:11:44,984 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:11:44.985 INFO:teuthology.orchestra.run.trial053.stderr:INFO 2026-02-26 03:11:44,984 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:11:44.987 INFO:teuthology.orchestra.run.trial053.stderr:==39591==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:11:45.047 INFO:teuthology.orchestra.run.trial053.stderr:DEBUG 2026-02-26 03:11:45,046 [shard 0:main] seastar - reactor::drain 2026-02-26T03:11:47.094 INFO:teuthology.orchestra.run.trial053.stderr:----------------------------------------------------- 2026-02-26T03:11:47.094 INFO:teuthology.orchestra.run.trial053.stderr:Suppressions used: 2026-02-26T03:11:47.094 INFO:teuthology.orchestra.run.trial053.stderr: count bytes template 2026-02-26T03:11:47.094 INFO:teuthology.orchestra.run.trial053.stderr: 1 8 MallocExtension::Register 2026-02-26T03:11:47.095 INFO:teuthology.orchestra.run.trial053.stderr:----------------------------------------------------- 2026-02-26T03:11:47.095 INFO:teuthology.orchestra.run.trial053.stderr: 2026-02-26T03:11:47.113 INFO:teuthology.orchestra.run.trial053.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-26T03:11:47.114 INFO:teuthology.orchestra.run.trial053.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-26T03:11:47.155 INFO:teuthology.orchestra.run.trial053.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-26T03:11:47.156 INFO:teuthology.orchestra.run.trial053.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-26T03:11:47.162 INFO:teuthology.orchestra.run.trial053.stderr:INFO 2026-02-26 03:11:47,161 seastar - Reactor backend: linux-aio 2026-02-26T03:11:47.176 INFO:teuthology.orchestra.run.trial053.stderr:INFO 2026-02-26 03:11:47,176 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:11:47.177 INFO:teuthology.orchestra.run.trial053.stderr:INFO 2026-02-26 03:11:47,176 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:11:47.180 INFO:teuthology.orchestra.run.trial053.stderr:==39589==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:11:47.180 INFO:teuthology.orchestra.run.trial053.stderr:INFO 2026-02-26 03:11:47,179 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-26T03:11:47.321 INFO:teuthology.orchestra.run.trial053.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x9c5139f 0x9c51a44 0x9c59eeb 0x9c5a5cb 0x9d43118 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:11:47.321 INFO:teuthology.orchestra.run.trial053.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0xe069f5 0x11253d1 0xf2e0d1 0x9c50a8f 0x9c50f59 0x9c511cc 0x9c52ef3 0x9c53440 0x9c5363f 0x9d38961 0x9d390bc 0x9d4334a 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:11:47.321 INFO:teuthology.orchestra.run.trial053.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x35ed822 0x35ed86b 0x36096e8 0x9c63157 0x9c6325f 0x9d23007 0x9d2319c 0x9d2352e 0x9d24b15 0x9d25e30 0x9d26250 0x9d42adb 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:11:47.468 INFO:teuthology.orchestra.run.trial053.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0xb75c8b 0xbb9340 0xe05f80 0x7c933ed 0x7bdaa69 0x7be9ecd 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:11:47.468 INFO:teuthology.orchestra.run.trial053.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc13 /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15e37 0x9c50e1a 0x9c511cc 0x9c52ef3 0x9c53440 0x9c5363f 0x9d38961 0x9d390bc 0x9d4334a 0x9d477c3 0x9d07d91 0x9d06d91 0x7beb0f2 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:11:47.468 INFO:teuthology.orchestra.run.trial053.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x15e88 0x9c50e1a 0x9c511cc 0x9c51482 0x9c51a44 0x9c59eeb 0x9c5a5cb 0x9d43118 0x9d477c3 0x9d07d91 0x9d06d91 0x7beb0f2 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:11:49.602 INFO:teuthology.orchestra.run.trial053.stderr:----------------------------------------------------- 2026-02-26T03:11:49.602 INFO:teuthology.orchestra.run.trial053.stderr:Suppressions used: 2026-02-26T03:11:49.602 INFO:teuthology.orchestra.run.trial053.stderr: count bytes template 2026-02-26T03:11:49.603 INFO:teuthology.orchestra.run.trial053.stderr: 1 8 MallocExtension::Register 2026-02-26T03:11:49.603 INFO:teuthology.orchestra.run.trial053.stderr:----------------------------------------------------- 2026-02-26T03:11:49.603 INFO:teuthology.orchestra.run.trial053.stderr: 2026-02-26T03:11:49.633 DEBUG:teuthology.orchestra.run.trial053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-26T03:11:49.659 DEBUG:teuthology.orchestra.run.trial053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:11:49.810 INFO:teuthology.orchestra.run.trial053.stderr:INFO  osd - OSD::main: parsing early config 2026-02-26T03:11:49.863 INFO:teuthology.orchestra.run.trial053.stderr:INFO 2026-02-26 03:11:49,862 seastar - Reactor backend: linux-aio 2026-02-26T03:11:49.876 INFO:teuthology.orchestra.run.trial053.stderr:INFO 2026-02-26 03:11:49,875 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:11:49.877 INFO:teuthology.orchestra.run.trial053.stderr:INFO 2026-02-26 03:11:49,876 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:11:49.879 INFO:teuthology.orchestra.run.trial053.stderr:==39657==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:11:49.938 INFO:teuthology.orchestra.run.trial053.stderr:DEBUG 2026-02-26 03:11:49,937 [shard 0:main] seastar - reactor::drain 2026-02-26T03:11:51.980 INFO:teuthology.orchestra.run.trial053.stderr:----------------------------------------------------- 2026-02-26T03:11:51.980 INFO:teuthology.orchestra.run.trial053.stderr:Suppressions used: 2026-02-26T03:11:51.980 INFO:teuthology.orchestra.run.trial053.stderr: count bytes template 2026-02-26T03:11:51.981 INFO:teuthology.orchestra.run.trial053.stderr: 1 8 MallocExtension::Register 2026-02-26T03:11:51.981 INFO:teuthology.orchestra.run.trial053.stderr:----------------------------------------------------- 2026-02-26T03:11:51.981 INFO:teuthology.orchestra.run.trial053.stderr: 2026-02-26T03:11:52.000 INFO:teuthology.orchestra.run.trial053.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-26T03:11:52.000 INFO:teuthology.orchestra.run.trial053.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-26T03:11:52.043 INFO:teuthology.orchestra.run.trial053.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-26T03:11:52.044 INFO:teuthology.orchestra.run.trial053.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-26T03:11:52.050 INFO:teuthology.orchestra.run.trial053.stderr:INFO 2026-02-26 03:11:52,049 seastar - Reactor backend: linux-aio 2026-02-26T03:11:52.062 INFO:teuthology.orchestra.run.trial053.stderr:INFO 2026-02-26 03:11:52,061 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:11:52.062 INFO:teuthology.orchestra.run.trial053.stderr:INFO 2026-02-26 03:11:52,061 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:11:52.066 INFO:teuthology.orchestra.run.trial053.stderr:==39655==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:11:52.066 INFO:teuthology.orchestra.run.trial053.stderr:INFO 2026-02-26 03:11:52,065 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-26T03:11:52.206 INFO:teuthology.orchestra.run.trial053.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1583b 0x10908ef 0xf2e0dc 0x9d396b3 0x9d3ebdc 0x9d3f21a 0x9d496e7 0x78947dd 0x7bea228 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:11:52.207 INFO:teuthology.orchestra.run.trial053.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x748c80d 0x73d690a 0x7159a3e 0x7be9f75 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:11:52.207 INFO:teuthology.orchestra.run.trial053.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x341f /lib64/libasan.so.8+0x15197 0x1125428 0xf2e0d1 0x9c50c5c 0x9c50f59 0x9c511cc 0x9c51482 0x9c51a44 0x9c59eeb 0x9c5a5cb 0x9d43118 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:11:52.358 INFO:teuthology.orchestra.run.trial053.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc08 /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x1125428 0xf2e0d1 0x9c5477a 0x9c54a27 0x9d4998d 0x78947dd 0x7bea228 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:11:52.359 INFO:teuthology.orchestra.run.trial053.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x9c50bc1 0x9c50f59 0x9c511cc 0x9c51482 0x9c51a44 0x9c59eeb 0x9c5a5cb 0x9d43118 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:11:52.359 INFO:teuthology.orchestra.run.trial053.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297db /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15e37 0x9c50e1a 0x9c511cc 0x9c51482 0x9c51a44 0x9c59eeb 0x9c5a5cb 0x9d43118 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:11:54.483 INFO:teuthology.orchestra.run.trial053.stderr:----------------------------------------------------- 2026-02-26T03:11:54.483 INFO:teuthology.orchestra.run.trial053.stderr:Suppressions used: 2026-02-26T03:11:54.483 INFO:teuthology.orchestra.run.trial053.stderr: count bytes template 2026-02-26T03:11:54.483 INFO:teuthology.orchestra.run.trial053.stderr: 1 8 MallocExtension::Register 2026-02-26T03:11:54.484 INFO:teuthology.orchestra.run.trial053.stderr:----------------------------------------------------- 2026-02-26T03:11:54.484 INFO:teuthology.orchestra.run.trial053.stderr: 2026-02-26T03:11:54.514 DEBUG:teuthology.orchestra.run.trial053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-26T03:11:54.539 DEBUG:teuthology.orchestra.run.trial053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:11:54.690 INFO:teuthology.orchestra.run.trial053.stderr:INFO  osd - OSD::main: parsing early config 2026-02-26T03:11:54.746 INFO:teuthology.orchestra.run.trial053.stderr:INFO 2026-02-26 03:11:54,745 seastar - Reactor backend: linux-aio 2026-02-26T03:11:54.755 INFO:teuthology.orchestra.run.trial053.stderr:INFO 2026-02-26 03:11:54,755 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:11:54.756 INFO:teuthology.orchestra.run.trial053.stderr:INFO 2026-02-26 03:11:54,755 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:11:54.758 INFO:teuthology.orchestra.run.trial053.stderr:==39724==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:11:54.817 INFO:teuthology.orchestra.run.trial053.stderr:DEBUG 2026-02-26 03:11:54,816 [shard 0:main] seastar - reactor::drain 2026-02-26T03:11:56.863 INFO:teuthology.orchestra.run.trial053.stderr:----------------------------------------------------- 2026-02-26T03:11:56.863 INFO:teuthology.orchestra.run.trial053.stderr:Suppressions used: 2026-02-26T03:11:56.863 INFO:teuthology.orchestra.run.trial053.stderr: count bytes template 2026-02-26T03:11:56.863 INFO:teuthology.orchestra.run.trial053.stderr: 1 8 MallocExtension::Register 2026-02-26T03:11:56.863 INFO:teuthology.orchestra.run.trial053.stderr:----------------------------------------------------- 2026-02-26T03:11:56.863 INFO:teuthology.orchestra.run.trial053.stderr: 2026-02-26T03:11:56.882 INFO:teuthology.orchestra.run.trial053.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-26T03:11:56.882 INFO:teuthology.orchestra.run.trial053.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-26T03:11:56.924 INFO:teuthology.orchestra.run.trial053.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-26T03:11:56.924 INFO:teuthology.orchestra.run.trial053.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-26T03:11:56.931 INFO:teuthology.orchestra.run.trial053.stderr:INFO 2026-02-26 03:11:56,930 seastar - Reactor backend: linux-aio 2026-02-26T03:11:56.939 INFO:teuthology.orchestra.run.trial053.stderr:INFO 2026-02-26 03:11:56,938 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:11:56.939 INFO:teuthology.orchestra.run.trial053.stderr:INFO 2026-02-26 03:11:56,938 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:11:56.942 INFO:teuthology.orchestra.run.trial053.stderr:==39722==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:11:56.942 INFO:teuthology.orchestra.run.trial053.stderr:INFO 2026-02-26 03:11:56,942 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-26T03:11:57.081 INFO:teuthology.orchestra.run.trial053.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1583b 0x7bdaa78 0x7be9ecd 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:11:57.081 INFO:teuthology.orchestra.run.trial053.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x10068f /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x157c7 0x10908ef 0xf2e0dc 0x9d33899 0x9d33e86 0x9d34209 0x9d44360 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:11:57.082 INFO:teuthology.orchestra.run.trial053.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc59b /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9b2559b 0x9ba2d36 0x9ba2e4c 0x9ba2f28 0x9ba31a4 0x9ba374b 0x9ba3b4f 0x9ba3e0c 0x9d45991 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:11:57.225 INFO:teuthology.orchestra.run.trial053.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x11904b /lib64/libasan.so.8+0xebbf /lib64/libasan.so.8+0xef905 0x31724fc 0x31726d6 0x317298d 0x3172a82 0x9d45d59 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:11:57.226 INFO:teuthology.orchestra.run.trial053.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x15838 0xe3f87b 0x3389298 0x9d08e54 0x9d49b8c 0x78947dd 0x7beb047 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:11:57.226 INFO:teuthology.orchestra.run.trial053.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x191c1b /lib64/libasan.so.8+0xc748 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9b2559b 0x9ba2d36 0x9ba2e4c 0x9ba2f28 0x9ba31a4 0x9ba374b 0x9ba3b4f 0x9ba3e0c 0x9ba4c54 0x9d1b824 0x9d1b91d 0x7be9c60 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:11:57.226 INFO:teuthology.orchestra.run.trial053.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff9672a157 0xffffffff963f9006 0xffffffff963fc9c0 0xffffffff963fdd21 0xffffffff96425a53 0xffffffff96425c15 0xffffffff963cc8ce 0xffffffff963d4bab 0xffffffff963d500b 0xffffffff960910d2 0xffffffff96d50451 0xffffffff96e00ba2 2026-02-26T03:11:59.359 INFO:teuthology.orchestra.run.trial053.stderr:----------------------------------------------------- 2026-02-26T03:11:59.359 INFO:teuthology.orchestra.run.trial053.stderr:Suppressions used: 2026-02-26T03:11:59.359 INFO:teuthology.orchestra.run.trial053.stderr: count bytes template 2026-02-26T03:11:59.359 INFO:teuthology.orchestra.run.trial053.stderr: 1 8 MallocExtension::Register 2026-02-26T03:11:59.359 INFO:teuthology.orchestra.run.trial053.stderr:----------------------------------------------------- 2026-02-26T03:11:59.359 INFO:teuthology.orchestra.run.trial053.stderr: 2026-02-26T03:11:59.391 DEBUG:teuthology.orchestra.run.trial053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-26T03:11:59.416 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-26T03:11:59.417 DEBUG:teuthology.orchestra.run.trial053:> set -ex 2026-02-26T03:11:59.417 DEBUG:teuthology.orchestra.run.trial053:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-26T03:11:59.480 DEBUG:teuthology.orchestra.run.trial053:> set -ex 2026-02-26T03:11:59.480 DEBUG:teuthology.orchestra.run.trial053:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-26T03:11:59.545 DEBUG:teuthology.orchestra.run.trial053:> set -ex 2026-02-26T03:11:59.545 DEBUG:teuthology.orchestra.run.trial053:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-26T03:11:59.608 DEBUG:teuthology.orchestra.run.trial053:> set -ex 2026-02-26T03:11:59.609 DEBUG:teuthology.orchestra.run.trial053:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-26T03:11:59.672 DEBUG:teuthology.orchestra.run.trial053:> set -ex 2026-02-26T03:11:59.672 DEBUG:teuthology.orchestra.run.trial053:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-26T03:11:59.727 INFO:tasks.ceph:Adding keys to all mons... 2026-02-26T03:11:59.727 DEBUG:teuthology.orchestra.run.trial053:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-26T03:11:59.789 INFO:teuthology.orchestra.run.trial053.stdout:[mgr.x] 2026-02-26T03:11:59.790 INFO:teuthology.orchestra.run.trial053.stdout: key = AQDuuZ9pgx2xBhAAJceZVOSoilf1bPNxHdDY0w== 2026-02-26T03:11:59.790 INFO:teuthology.orchestra.run.trial053.stdout:[osd.0] 2026-02-26T03:11:59.790 INFO:teuthology.orchestra.run.trial053.stdout: key = AQDzuZ9pHxOUDhAA6TwU+95t4MrqrRgII4VydA== 2026-02-26T03:11:59.790 INFO:teuthology.orchestra.run.trial053.stdout:[osd.1] 2026-02-26T03:11:59.790 INFO:teuthology.orchestra.run.trial053.stdout: key = AQD4uZ9poLvLBxAAalqZJJ5q/0msudt7dXSfqQ== 2026-02-26T03:11:59.790 INFO:teuthology.orchestra.run.trial053.stdout:[osd.2] 2026-02-26T03:11:59.790 INFO:teuthology.orchestra.run.trial053.stdout: key = AQD9uZ9pc61fABAAqfxzVMpUrFcAsAKF99jRVQ== 2026-02-26T03:11:59.790 INFO:teuthology.orchestra.run.trial053.stdout:[client.0] 2026-02-26T03:11:59.790 INFO:teuthology.orchestra.run.trial053.stdout: key = AQDuuZ9pw6xeCRAAr8Fxc8wS8j83qEUW579A0w== 2026-02-26T03:11:59.791 DEBUG:teuthology.orchestra.run.trial053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-26T03:11:59.879 DEBUG:teuthology.orchestra.run.trial053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-26T03:11:59.928 DEBUG:teuthology.orchestra.run.trial053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-26T03:11:59.977 DEBUG:teuthology.orchestra.run.trial053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-26T03:12:00.024 DEBUG:teuthology.orchestra.run.trial053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-26T03:12:00.075 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-26T03:12:00.075 DEBUG:teuthology.orchestra.run.trial053:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-26T03:12:00.100 DEBUG:teuthology.orchestra.run.trial053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-26T03:12:00.186 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:00.184+0000 7f956de87c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:00.187 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:00.184+0000 7f956de87c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:00.194 DEBUG:teuthology.orchestra.run.trial053:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-26T03:12:00.219 DEBUG:teuthology.orchestra.run.trial053:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:12:00.275 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-26T03:12:00.275 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-26T03:12:00.275 DEBUG:teuthology.orchestra.run.trial053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-26T03:12:00.317 INFO:tasks.ceph.mon.a:Started 2026-02-26T03:12:00.318 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-26T03:12:00.318 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-26T03:12:00.318 DEBUG:teuthology.orchestra.run.trial053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-26T03:12:00.320 INFO:tasks.ceph.mgr.x:Started 2026-02-26T03:12:00.321 DEBUG:tasks.ceph:set 0 configs 2026-02-26T03:12:00.321 DEBUG:teuthology.orchestra.run.trial053:> sudo ceph --cluster ceph config dump 2026-02-26T03:12:00.380 INFO:tasks.ceph.mon.a.trial053.stderr:2026-02-26T03:12:00.377+0000 7f8094a30c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:00.380 INFO:tasks.ceph.mon.a.trial053.stderr:2026-02-26T03:12:00.377+0000 7f8094a30c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:00.380 INFO:tasks.ceph.mon.a.trial053.stderr:2026-02-26T03:12:00.377+0000 7f8094a30c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:00.384 INFO:tasks.ceph.mgr.x.trial053.stderr:2026-02-26T03:12:00.382+0000 7f7d53bc0fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:00.396 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:00.392+0000 7fd371ffd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:00.397 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:00.395+0000 7fd371ffd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:00.429 INFO:teuthology.orchestra.run.trial053.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-26T03:12:00.436 INFO:tasks.ceph:Setting crush tunables to default 2026-02-26T03:12:00.436 DEBUG:teuthology.orchestra.run.trial053:> sudo ceph --cluster ceph osd crush tunables default 2026-02-26T03:12:00.513 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:00.508+0000 7f9f0270b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:00.514 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:00.512+0000 7f9f0270b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:00.547 INFO:teuthology.orchestra.run.trial053.stderr:adjusted tunables profile to default 2026-02-26T03:12:00.555 INFO:tasks.ceph:check_enable_crimson: True 2026-02-26T03:12:00.556 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2026-02-26T03:12:00.556 DEBUG:teuthology.orchestra.run.trial053:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-26T03:12:00.586 INFO:tasks.ceph.mgr.x.trial053.stderr:2026-02-26T03:12:00.584+0000 7f7d53bc0fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:00.586 INFO:tasks.ceph.mgr.x.trial053.stderr:2026-02-26T03:12:00.584+0000 7f7d53bc0fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:00.630 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:00.626+0000 7f4ef5ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:00.632 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:00.629+0000 7f4ef5ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:00.670 INFO:tasks.ceph.mon.a.trial053.stderr:2026-02-26T03:12:00.668+0000 7f808bdf1640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-26T03:12:00.670 INFO:tasks.ceph.mon.a.trial053.stderr:Please be aware that this feature is experimental, untested, 2026-02-26T03:12:00.670 INFO:tasks.ceph.mon.a.trial053.stderr:unsupported, and may result in data corruption, data loss, 2026-02-26T03:12:00.670 INFO:tasks.ceph.mon.a.trial053.stderr:and/or irreparable damage to your cluster. Do not use 2026-02-26T03:12:00.671 INFO:tasks.ceph.mon.a.trial053.stderr:feature with important data. 2026-02-26T03:12:00.671 INFO:tasks.ceph.mon.a.trial053.stderr: 2026-02-26T03:12:01.430 INFO:tasks.ceph.mgr.x.trial053.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-26T03:12:01.431 INFO:tasks.ceph.mgr.x.trial053.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-26T03:12:01.431 INFO:tasks.ceph.mgr.x.trial053.stderr: from numpy import show_config as show_numpy_config 2026-02-26T03:12:01.558 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-26T03:12:01.558 DEBUG:teuthology.orchestra.run.trial053:> set -ex 2026-02-26T03:12:01.559 DEBUG:teuthology.orchestra.run.trial053:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-26T03:12:01.621 DEBUG:teuthology.orchestra.run.trial053:> set -ex 2026-02-26T03:12:01.621 DEBUG:teuthology.orchestra.run.trial053:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-26T03:12:01.686 DEBUG:teuthology.orchestra.run.trial053:> set -ex 2026-02-26T03:12:01.686 DEBUG:teuthology.orchestra.run.trial053:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-26T03:12:01.752 DEBUG:teuthology.orchestra.run.trial053:> sudo ceph --cluster ceph osd new f81b2eff-cd91-48b3-b727-234254c99354 0 2026-02-26T03:12:01.865 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:01.860+0000 7f308ebb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:01.866 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:01.864+0000 7f308ebb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:01.903 INFO:teuthology.orchestra.run.trial053.stdout:0 2026-02-26T03:12:01.911 DEBUG:teuthology.orchestra.run.trial053:> sudo ceph --cluster ceph osd new aae88a13-cdce-42cf-9c57-05336b14fba7 1 2026-02-26T03:12:01.984 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:01.979+0000 7f15f1c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:01.985 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:01.984+0000 7f15f09d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:02.021 INFO:teuthology.orchestra.run.trial053.stdout:1 2026-02-26T03:12:02.028 DEBUG:teuthology.orchestra.run.trial053:> sudo ceph --cluster ceph osd new 40962bbd-49d0-4529-b050-e46412f3642c 2 2026-02-26T03:12:02.101 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:02.097+0000 7f0e75e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:02.102 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:02.101+0000 7f0e75e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:02.139 INFO:teuthology.orchestra.run.trial053.stdout:2 2026-02-26T03:12:02.147 DEBUG:teuthology.orchestra.run.trial053:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2026-02-26T03:12:02.175 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-26T03:12:02.175 DEBUG:teuthology.orchestra.run.trial053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-26T03:12:02.217 INFO:tasks.ceph.osd.0:Started 2026-02-26T03:12:02.217 DEBUG:teuthology.orchestra.run.trial053:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2026-02-26T03:12:02.245 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-26T03:12:02.245 DEBUG:teuthology.orchestra.run.trial053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-26T03:12:02.287 INFO:tasks.ceph.osd.1:Started 2026-02-26T03:12:02.287 DEBUG:teuthology.orchestra.run.trial053:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2026-02-26T03:12:02.316 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-26T03:12:02.316 DEBUG:teuthology.orchestra.run.trial053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-26T03:12:02.358 INFO:tasks.ceph.osd.2:Started 2026-02-26T03:12:02.358 DEBUG:teuthology.orchestra.run.trial053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T03:12:02.434 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:02.430+0000 7fbc6ef74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:02.435 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:02.433+0000 7fbc6ef74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:02.468 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:12:02.469 INFO:teuthology.orchestra.run.trial053.stdout:{"epoch":6,"fsid":"10bcb95a-6ed5-4257-8ca2-306c33baed91","created":"2026-02-26T03:12:00.390414+0000","modified":"2026-02-26T03:12:02.137589+0000","last_up_change":"0.000000","last_in_change":"2026-02-26T03:12:02.137589+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"f81b2eff-cd91-48b3-b727-234254c99354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"aae88a13-cdce-42cf-9c57-05336b14fba7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"40962bbd-49d0-4529-b050-e46412f3642c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:12:02.477 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-26T03:12:02.477 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-26T03:12:02.779 DEBUG:teuthology.orchestra.run.trial053:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-26T03:12:02.846 INFO:teuthology.misc.health.trial053.stderr:2026-02-26T03:12:02.843+0000 7f310d364640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:02.848 INFO:teuthology.misc.health.trial053.stderr:2026-02-26T03:12:02.846+0000 7f310d364640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:02.880 INFO:teuthology.misc.health.trial053.stdout: 2026-02-26T03:12:02.880 INFO:teuthology.misc.health.trial053.stdout:{"epoch":6,"fsid":"10bcb95a-6ed5-4257-8ca2-306c33baed91","created":"2026-02-26T03:12:00.390414+0000","modified":"2026-02-26T03:12:02.137589+0000","last_up_change":"0.000000","last_in_change":"2026-02-26T03:12:02.137589+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"f81b2eff-cd91-48b3-b727-234254c99354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"aae88a13-cdce-42cf-9c57-05336b14fba7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"40962bbd-49d0-4529-b050-e46412f3642c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:12:02.888 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-26T03:12:03.832 INFO:tasks.ceph.mgr.x.trial053.stderr:2026-02-26T03:12:03.830+0000 7f7c6eb03640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-26T03:12:09.190 DEBUG:teuthology.orchestra.run.trial053:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-26T03:12:09.262 INFO:teuthology.misc.health.trial053.stderr:2026-02-26T03:12:09.258+0000 7f5a943fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:09.263 INFO:teuthology.misc.health.trial053.stderr:2026-02-26T03:12:09.261+0000 7f5a943fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:09.376 INFO:teuthology.misc.health.trial053.stdout: 2026-02-26T03:12:09.376 INFO:teuthology.misc.health.trial053.stdout:{"epoch":8,"fsid":"10bcb95a-6ed5-4257-8ca2-306c33baed91","created":"2026-02-26T03:12:00.390414+0000","modified":"2026-02-26T03:12:09.256404+0000","last_up_change":"0.000000","last_in_change":"2026-02-26T03:12:02.137589+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"f81b2eff-cd91-48b3-b727-234254c99354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"aae88a13-cdce-42cf-9c57-05336b14fba7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"40962bbd-49d0-4529-b050-e46412f3642c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:12:09.385 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-26T03:12:15.688 DEBUG:teuthology.orchestra.run.trial053:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-26T03:12:15.754 INFO:teuthology.misc.health.trial053.stderr:2026-02-26T03:12:15.750+0000 7f0c644cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:15.755 INFO:teuthology.misc.health.trial053.stderr:2026-02-26T03:12:15.753+0000 7f0c644cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:15.864 INFO:teuthology.misc.health.trial053.stdout: 2026-02-26T03:12:15.864 INFO:teuthology.misc.health.trial053.stdout:{"epoch":13,"fsid":"10bcb95a-6ed5-4257-8ca2-306c33baed91","created":"2026-02-26T03:12:00.390414+0000","modified":"2026-02-26T03:12:15.274765+0000","last_up_change":"2026-02-26T03:12:11.263982+0000","last_in_change":"2026-02-26T03:12:02.137589+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-26T03:12:12.841391+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f81b2eff-cd91-48b3-b727-234254c99354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6806","nonce":104342457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6807","nonce":104342457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6812","nonce":104342457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6813","nonce":104342457}]},"public_addr":"10.20.193.53:6806/104342457","cluster_addr":"10.20.193.53:6807/104342457","heartbeat_back_addr":"10.20.193.53:6812/104342457","heartbeat_front_addr":"10.20.193.53:6813/104342457","state":["exists","up"]},{"osd":1,"uuid":"aae88a13-cdce-42cf-9c57-05336b14fba7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6805","nonce":2670331724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6802","nonce":2670331724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6811","nonce":2670331724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6810","nonce":2670331724}]},"public_addr":"10.20.193.53:6805/2670331724","cluster_addr":"10.20.193.53:6802/2670331724","heartbeat_back_addr":"10.20.193.53:6811/2670331724","heartbeat_front_addr":"10.20.193.53:6810/2670331724","state":["exists","up"]},{"osd":2,"uuid":"40962bbd-49d0-4529-b050-e46412f3642c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6804","nonce":1907649219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6803","nonce":1907649219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6808","nonce":1907649219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6809","nonce":1907649219}]},"public_addr":"10.20.193.53:6804/1907649219","cluster_addr":"10.20.193.53:6803/1907649219","heartbeat_back_addr":"10.20.193.53:6808/1907649219","heartbeat_front_addr":"10.20.193.53:6809/1907649219","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:12:15.871 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-26T03:12:15.871 INFO:tasks.ceph:Creating RBD pool 2026-02-26T03:12:15.872 DEBUG:teuthology.orchestra.run.trial053:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-26T03:12:15.945 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:15.940+0000 7f1d5c55c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:15.947 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:15.945+0000 7f1d5c55c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:16.603 INFO:teuthology.orchestra.run.trial053.stderr:pool 'rbd' created 2026-02-26T03:12:16.617 DEBUG:teuthology.orchestra.run.trial053:> rbd --cluster ceph pool init rbd 2026-02-26T03:12:16.658 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:16.656+0000 7f88160df240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:16.659 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:16.656+0000 7f88160df240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:16.660 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:16.658+0000 7f88160df240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:19.638 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-26T03:12:19.639 DEBUG:teuthology.orchestra.run.trial053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-26T03:12:19.639 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-26T03:12:19.754 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:19.749+0000 7fe43c41a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:19.755 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:19.753+0000 7fe43c41a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:19.863 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:12:19.872 INFO:teuthology.orchestra.run.trial053.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-26T03:12:19.872 INFO:tasks.ceph_manager:config epoch is 1 2026-02-26T03:12:19.872 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-26T03:12:19.873 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-26T03:12:19.873 DEBUG:teuthology.orchestra.run.trial053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-26T03:12:19.948 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:19.944+0000 7fa0deb0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:19.949 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:19.947+0000 7fa0deb0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:20.093 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:12:20.103 INFO:teuthology.orchestra.run.trial053.stdout:{"epoch":5,"flags":0,"active_gid":4103,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6800","nonce":2188664866},{"type":"v1","addr":"10.20.193.53:6801","nonce":2188664866}]},"active_addr":"10.20.193.53:6801/2188664866","active_change":"2026-02-26T03:12:02.826097+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.53:0","nonce":3473232352}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.53:0","nonce":3322933171}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.53:0","nonce":1743900190}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.53:0","nonce":1437030537}]}]} 2026-02-26T03:12:20.106 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-26T03:12:20.107 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-26T03:12:20.107 DEBUG:teuthology.orchestra.run.trial053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T03:12:20.186 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:20.182+0000 7fabd770b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:20.187 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:20.185+0000 7fabd770b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:20.297 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:12:20.297 INFO:teuthology.orchestra.run.trial053.stdout:{"epoch":17,"fsid":"10bcb95a-6ed5-4257-8ca2-306c33baed91","created":"2026-02-26T03:12:00.390414+0000","modified":"2026-02-26T03:12:19.611270+0000","last_up_change":"2026-02-26T03:12:11.263982+0000","last_in_change":"2026-02-26T03:12:02.137589+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-26T03:12:12.841391+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-26T03:12:16.057019+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f81b2eff-cd91-48b3-b727-234254c99354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6806","nonce":104342457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6807","nonce":104342457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6812","nonce":104342457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6813","nonce":104342457}]},"public_addr":"10.20.193.53:6806/104342457","cluster_addr":"10.20.193.53:6807/104342457","heartbeat_back_addr":"10.20.193.53:6812/104342457","heartbeat_front_addr":"10.20.193.53:6813/104342457","state":["exists","up"]},{"osd":1,"uuid":"aae88a13-cdce-42cf-9c57-05336b14fba7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6805","nonce":2670331724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6802","nonce":2670331724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6811","nonce":2670331724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6810","nonce":2670331724}]},"public_addr":"10.20.193.53:6805/2670331724","cluster_addr":"10.20.193.53:6802/2670331724","heartbeat_back_addr":"10.20.193.53:6811/2670331724","heartbeat_front_addr":"10.20.193.53:6810/2670331724","state":["exists","up"]},{"osd":2,"uuid":"40962bbd-49d0-4529-b050-e46412f3642c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6804","nonce":1907649219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6803","nonce":1907649219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6808","nonce":1907649219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6809","nonce":1907649219}]},"public_addr":"10.20.193.53:6804/1907649219","cluster_addr":"10.20.193.53:6803/1907649219","heartbeat_back_addr":"10.20.193.53:6808/1907649219","heartbeat_front_addr":"10.20.193.53:6809/1907649219","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:12:20.307 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-26T03:12:20.308 DEBUG:teuthology.orchestra.run.trial053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T03:12:20.385 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:20.380+0000 7f0bdd55b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:20.386 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:20.383+0000 7f0bdd55b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:20.495 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:12:20.495 INFO:teuthology.orchestra.run.trial053.stdout:{"epoch":17,"fsid":"10bcb95a-6ed5-4257-8ca2-306c33baed91","created":"2026-02-26T03:12:00.390414+0000","modified":"2026-02-26T03:12:19.611270+0000","last_up_change":"2026-02-26T03:12:11.263982+0000","last_in_change":"2026-02-26T03:12:02.137589+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-26T03:12:12.841391+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-26T03:12:16.057019+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f81b2eff-cd91-48b3-b727-234254c99354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6806","nonce":104342457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6807","nonce":104342457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6812","nonce":104342457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6813","nonce":104342457}]},"public_addr":"10.20.193.53:6806/104342457","cluster_addr":"10.20.193.53:6807/104342457","heartbeat_back_addr":"10.20.193.53:6812/104342457","heartbeat_front_addr":"10.20.193.53:6813/104342457","state":["exists","up"]},{"osd":1,"uuid":"aae88a13-cdce-42cf-9c57-05336b14fba7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6805","nonce":2670331724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6802","nonce":2670331724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6811","nonce":2670331724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6810","nonce":2670331724}]},"public_addr":"10.20.193.53:6805/2670331724","cluster_addr":"10.20.193.53:6802/2670331724","heartbeat_back_addr":"10.20.193.53:6811/2670331724","heartbeat_front_addr":"10.20.193.53:6810/2670331724","state":["exists","up"]},{"osd":2,"uuid":"40962bbd-49d0-4529-b050-e46412f3642c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6804","nonce":1907649219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6803","nonce":1907649219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6808","nonce":1907649219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.53:6809","nonce":1907649219}]},"public_addr":"10.20.193.53:6804/1907649219","cluster_addr":"10.20.193.53:6803/1907649219","heartbeat_back_addr":"10.20.193.53:6808/1907649219","heartbeat_front_addr":"10.20.193.53:6809/1907649219","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:12:20.504 DEBUG:teuthology.orchestra.run.trial053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-26T03:12:20.505 DEBUG:teuthology.orchestra.run.trial053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-26T03:12:20.505 DEBUG:teuthology.orchestra.run.trial053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-26T03:12:20.585 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:20.580+0000 7f71b130b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:20.585 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:20.581+0000 7f9ded251640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:20.586 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:20.584+0000 7f71b130b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:20.586 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:20.584+0000 7f9ded251640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:20.588 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:20.583+0000 7f8a32e3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:20.589 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:20.588+0000 7f8a31bb4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:20.601 INFO:teuthology.orchestra.run.trial053.stdout:38654705674 2026-02-26T03:12:20.601 DEBUG:teuthology.orchestra.run.trial053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-26T03:12:20.602 INFO:teuthology.orchestra.run.trial053.stdout:42949672969 2026-02-26T03:12:20.602 DEBUG:teuthology.orchestra.run.trial053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-26T03:12:20.603 INFO:teuthology.orchestra.run.trial053.stdout:42949672969 2026-02-26T03:12:20.604 DEBUG:teuthology.orchestra.run.trial053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-26T03:12:20.722 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:20.717+0000 7f08505a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:20.723 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:20.718+0000 7f04f6c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:20.723 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:20.722+0000 7f08505a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:20.724 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:20.722+0000 7f04f6c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:20.765 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:20.761+0000 7f8193611640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:20.766 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:20.764+0000 7f8193611640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:20.836 INFO:teuthology.orchestra.run.trial053.stdout:38654705674 2026-02-26T03:12:20.838 INFO:teuthology.orchestra.run.trial053.stdout:42949672969 2026-02-26T03:12:20.847 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705674 for osd.2 2026-02-26T03:12:20.848 DEBUG:teuthology.parallel:result is None 2026-02-26T03:12:20.850 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672969 for osd.1 2026-02-26T03:12:20.850 DEBUG:teuthology.parallel:result is None 2026-02-26T03:12:20.876 INFO:teuthology.orchestra.run.trial053.stdout:42949672969 2026-02-26T03:12:20.886 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672969 for osd.0 2026-02-26T03:12:20.886 DEBUG:teuthology.parallel:result is None 2026-02-26T03:12:20.887 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-26T03:12:20.887 DEBUG:teuthology.orchestra.run.trial053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:12:21.009 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:21.004+0000 7fd68505a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:21.010 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:21.008+0000 7fd68505a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:21.116 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:12:21.117 INFO:teuthology.orchestra.run.trial053.stderr:dumped all 2026-02-26T03:12:21.125 INFO:teuthology.orchestra.run.trial053.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-26T03:12:20.830102+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286352,"kb_used":62917052,"kb_used_data":62917052,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":94369300,"statfs":{"total":161061224448,"available":96634163200,"internally_reserved":0,"allocated":64427061248,"data_stored":64427061248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.232806"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:12:19.624110+0000","last_change":"2026-02-26T03:12:19.628067+0000","last_active":"2026-02-26T03:12:19.624110+0000","last_peered":"2026-02-26T03:12:19.624110+0000","last_clean":"2026-02-26T03:12:19.624110+0000","last_became_active":"2026-02-26T03:12:17.642398+0000","last_became_peered":"2026-02-26T03:12:17.642398+0000","last_unstale":"2026-02-26T03:12:19.624110+0000","last_undegraded":"2026-02-26T03:12:19.624110+0000","last_fullsized":"2026-02-26T03:12:19.624110+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_clean_scrub_stamp":"2026-02-26T03:12:16.601631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:12:19.625488+0000","last_change":"2026-02-26T03:12:19.627107+0000","last_active":"2026-02-26T03:12:19.625488+0000","last_peered":"2026-02-26T03:12:19.625488+0000","last_clean":"2026-02-26T03:12:19.625488+0000","last_became_active":"2026-02-26T03:12:17.637384+0000","last_became_peered":"2026-02-26T03:12:17.637384+0000","last_unstale":"2026-02-26T03:12:19.625488+0000","last_undegraded":"2026-02-26T03:12:19.625488+0000","last_fullsized":"2026-02-26T03:12:19.625488+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_clean_scrub_stamp":"2026-02-26T03:12:16.601631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:12:19.625751+0000","last_change":"2026-02-26T03:12:19.628706+0000","last_active":"2026-02-26T03:12:19.625751+0000","last_peered":"2026-02-26T03:12:19.625751+0000","last_clean":"2026-02-26T03:12:19.625751+0000","last_became_active":"2026-02-26T03:12:17.638661+0000","last_became_peered":"2026-02-26T03:12:17.638661+0000","last_unstale":"2026-02-26T03:12:19.625751+0000","last_undegraded":"2026-02-26T03:12:19.625751+0000","last_fullsized":"2026-02-26T03:12:19.625751+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_clean_scrub_stamp":"2026-02-26T03:12:16.601631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:12:19.625506+0000","last_change":"2026-02-26T03:12:19.627772+0000","last_active":"2026-02-26T03:12:19.625506+0000","last_peered":"2026-02-26T03:12:19.625506+0000","last_clean":"2026-02-26T03:12:19.625506+0000","last_became_active":"2026-02-26T03:12:17.642279+0000","last_became_peered":"2026-02-26T03:12:17.642279+0000","last_unstale":"2026-02-26T03:12:19.625506+0000","last_undegraded":"2026-02-26T03:12:19.625506+0000","last_fullsized":"2026-02-26T03:12:19.625506+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_clean_scrub_stamp":"2026-02-26T03:12:16.601631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:12:19.625037+0000","last_change":"2026-02-26T03:12:19.625037+0000","last_active":"2026-02-26T03:12:19.625037+0000","last_peered":"2026-02-26T03:12:19.625037+0000","last_clean":"2026-02-26T03:12:19.625037+0000","last_became_active":"2026-02-26T03:12:17.651667+0000","last_became_peered":"2026-02-26T03:12:17.651667+0000","last_unstale":"2026-02-26T03:12:19.625037+0000","last_undegraded":"2026-02-26T03:12:19.625037+0000","last_fullsized":"2026-02-26T03:12:19.625037+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_clean_scrub_stamp":"2026-02-26T03:12:16.601631+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:12:19.624225+0000","last_change":"2026-02-26T03:12:19.626323+0000","last_active":"2026-02-26T03:12:19.624225+0000","last_peered":"2026-02-26T03:12:19.624225+0000","last_clean":"2026-02-26T03:12:19.624225+0000","last_became_active":"2026-02-26T03:12:17.650410+0000","last_became_peered":"2026-02-26T03:12:17.650410+0000","last_unstale":"2026-02-26T03:12:19.624225+0000","last_undegraded":"2026-02-26T03:12:19.624225+0000","last_fullsized":"2026-02-26T03:12:19.624225+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_clean_scrub_stamp":"2026-02-26T03:12:16.601631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:12:19.622025+0000","last_change":"2026-02-26T03:12:19.624543+0000","last_active":"2026-02-26T03:12:19.622025+0000","last_peered":"2026-02-26T03:12:19.622025+0000","last_clean":"2026-02-26T03:12:19.622025+0000","last_became_active":"2026-02-26T03:12:17.653409+0000","last_became_peered":"2026-02-26T03:12:17.653409+0000","last_unstale":"2026-02-26T03:12:19.622025+0000","last_undegraded":"2026-02-26T03:12:19.622025+0000","last_fullsized":"2026-02-26T03:12:19.622025+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_clean_scrub_stamp":"2026-02-26T03:12:16.601631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:12:19.625818+0000","last_change":"2026-02-26T03:12:19.628928+0000","last_active":"2026-02-26T03:12:19.625818+0000","last_peered":"2026-02-26T03:12:19.625818+0000","last_clean":"2026-02-26T03:12:19.625818+0000","last_became_active":"2026-02-26T03:12:17.637650+0000","last_became_peered":"2026-02-26T03:12:17.637650+0000","last_unstale":"2026-02-26T03:12:19.625818+0000","last_undegraded":"2026-02-26T03:12:19.625818+0000","last_fullsized":"2026-02-26T03:12:19.625818+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_clean_scrub_stamp":"2026-02-26T03:12:16.601631+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:12:19.624009+0000","last_change":"2026-02-26T03:12:14.302732+0000","last_active":"2026-02-26T03:12:19.624009+0000","last_peered":"2026-02-26T03:12:19.624009+0000","last_clean":"2026-02-26T03:12:19.624009+0000","last_became_active":"2026-02-26T03:12:14.295553+0000","last_became_peered":"2026-02-26T03:12:14.295553+0000","last_unstale":"2026-02-26T03:12:19.624009+0000","last_undegraded":"2026-02-26T03:12:19.624009+0000","last_fullsized":"2026-02-26T03:12:19.624009+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:12:13.265848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:12:13.265848+0000","last_clean_scrub_stamp":"2026-02-26T03:12:13.265848+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96634163200,"internally_reserved":0,"allocated":64427061248,"data_stored":64427061248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972652,"kb_used_data":20972652,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456132,"statfs":{"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972580,"kb_used_data":20972580,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456204,"statfs":{"total":53687074816,"available":32211152896,"internally_reserved":0,"allocated":21475921920,"data_stored":21475921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971820,"kb_used_data":20971820,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456964,"statfs":{"total":53687074816,"available":32211931136,"internally_reserved":0,"allocated":21475143680,"data_stored":21475143680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32211152896,"internally_reserved":0,"allocated":21475921920,"data_stored":21475921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53687074816,"available":32211931136,"internally_reserved":0,"allocated":21475143680,"data_stored":21475143680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:12:21.127 DEBUG:teuthology.orchestra.run.trial053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:12:21.203 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:21.199+0000 7fc80de1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:21.204 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:21.202+0000 7fc80de1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:21.313 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:12:21.313 INFO:teuthology.orchestra.run.trial053.stderr:dumped all 2026-02-26T03:12:21.323 INFO:teuthology.orchestra.run.trial053.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-26T03:12:20.830102+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286352,"kb_used":62917052,"kb_used_data":62917052,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":94369300,"statfs":{"total":161061224448,"available":96634163200,"internally_reserved":0,"allocated":64427061248,"data_stored":64427061248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.232806"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:12:19.624110+0000","last_change":"2026-02-26T03:12:19.628067+0000","last_active":"2026-02-26T03:12:19.624110+0000","last_peered":"2026-02-26T03:12:19.624110+0000","last_clean":"2026-02-26T03:12:19.624110+0000","last_became_active":"2026-02-26T03:12:17.642398+0000","last_became_peered":"2026-02-26T03:12:17.642398+0000","last_unstale":"2026-02-26T03:12:19.624110+0000","last_undegraded":"2026-02-26T03:12:19.624110+0000","last_fullsized":"2026-02-26T03:12:19.624110+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_clean_scrub_stamp":"2026-02-26T03:12:16.601631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:12:19.625488+0000","last_change":"2026-02-26T03:12:19.627107+0000","last_active":"2026-02-26T03:12:19.625488+0000","last_peered":"2026-02-26T03:12:19.625488+0000","last_clean":"2026-02-26T03:12:19.625488+0000","last_became_active":"2026-02-26T03:12:17.637384+0000","last_became_peered":"2026-02-26T03:12:17.637384+0000","last_unstale":"2026-02-26T03:12:19.625488+0000","last_undegraded":"2026-02-26T03:12:19.625488+0000","last_fullsized":"2026-02-26T03:12:19.625488+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_clean_scrub_stamp":"2026-02-26T03:12:16.601631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:12:19.625751+0000","last_change":"2026-02-26T03:12:19.628706+0000","last_active":"2026-02-26T03:12:19.625751+0000","last_peered":"2026-02-26T03:12:19.625751+0000","last_clean":"2026-02-26T03:12:19.625751+0000","last_became_active":"2026-02-26T03:12:17.638661+0000","last_became_peered":"2026-02-26T03:12:17.638661+0000","last_unstale":"2026-02-26T03:12:19.625751+0000","last_undegraded":"2026-02-26T03:12:19.625751+0000","last_fullsized":"2026-02-26T03:12:19.625751+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_clean_scrub_stamp":"2026-02-26T03:12:16.601631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:12:19.625506+0000","last_change":"2026-02-26T03:12:19.627772+0000","last_active":"2026-02-26T03:12:19.625506+0000","last_peered":"2026-02-26T03:12:19.625506+0000","last_clean":"2026-02-26T03:12:19.625506+0000","last_became_active":"2026-02-26T03:12:17.642279+0000","last_became_peered":"2026-02-26T03:12:17.642279+0000","last_unstale":"2026-02-26T03:12:19.625506+0000","last_undegraded":"2026-02-26T03:12:19.625506+0000","last_fullsized":"2026-02-26T03:12:19.625506+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_clean_scrub_stamp":"2026-02-26T03:12:16.601631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:12:19.625037+0000","last_change":"2026-02-26T03:12:19.625037+0000","last_active":"2026-02-26T03:12:19.625037+0000","last_peered":"2026-02-26T03:12:19.625037+0000","last_clean":"2026-02-26T03:12:19.625037+0000","last_became_active":"2026-02-26T03:12:17.651667+0000","last_became_peered":"2026-02-26T03:12:17.651667+0000","last_unstale":"2026-02-26T03:12:19.625037+0000","last_undegraded":"2026-02-26T03:12:19.625037+0000","last_fullsized":"2026-02-26T03:12:19.625037+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_clean_scrub_stamp":"2026-02-26T03:12:16.601631+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:12:19.624225+0000","last_change":"2026-02-26T03:12:19.626323+0000","last_active":"2026-02-26T03:12:19.624225+0000","last_peered":"2026-02-26T03:12:19.624225+0000","last_clean":"2026-02-26T03:12:19.624225+0000","last_became_active":"2026-02-26T03:12:17.650410+0000","last_became_peered":"2026-02-26T03:12:17.650410+0000","last_unstale":"2026-02-26T03:12:19.624225+0000","last_undegraded":"2026-02-26T03:12:19.624225+0000","last_fullsized":"2026-02-26T03:12:19.624225+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_clean_scrub_stamp":"2026-02-26T03:12:16.601631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:12:19.622025+0000","last_change":"2026-02-26T03:12:19.624543+0000","last_active":"2026-02-26T03:12:19.622025+0000","last_peered":"2026-02-26T03:12:19.622025+0000","last_clean":"2026-02-26T03:12:19.622025+0000","last_became_active":"2026-02-26T03:12:17.653409+0000","last_became_peered":"2026-02-26T03:12:17.653409+0000","last_unstale":"2026-02-26T03:12:19.622025+0000","last_undegraded":"2026-02-26T03:12:19.622025+0000","last_fullsized":"2026-02-26T03:12:19.622025+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_clean_scrub_stamp":"2026-02-26T03:12:16.601631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:12:19.625818+0000","last_change":"2026-02-26T03:12:19.628928+0000","last_active":"2026-02-26T03:12:19.625818+0000","last_peered":"2026-02-26T03:12:19.625818+0000","last_clean":"2026-02-26T03:12:19.625818+0000","last_became_active":"2026-02-26T03:12:17.637650+0000","last_became_peered":"2026-02-26T03:12:17.637650+0000","last_unstale":"2026-02-26T03:12:19.625818+0000","last_undegraded":"2026-02-26T03:12:19.625818+0000","last_fullsized":"2026-02-26T03:12:19.625818+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:12:16.601631+0000","last_clean_scrub_stamp":"2026-02-26T03:12:16.601631+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:12:19.624009+0000","last_change":"2026-02-26T03:12:14.302732+0000","last_active":"2026-02-26T03:12:19.624009+0000","last_peered":"2026-02-26T03:12:19.624009+0000","last_clean":"2026-02-26T03:12:19.624009+0000","last_became_active":"2026-02-26T03:12:14.295553+0000","last_became_peered":"2026-02-26T03:12:14.295553+0000","last_unstale":"2026-02-26T03:12:19.624009+0000","last_undegraded":"2026-02-26T03:12:19.624009+0000","last_fullsized":"2026-02-26T03:12:19.624009+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:12:13.265848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:12:13.265848+0000","last_clean_scrub_stamp":"2026-02-26T03:12:13.265848+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96634163200,"internally_reserved":0,"allocated":64427061248,"data_stored":64427061248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972652,"kb_used_data":20972652,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456132,"statfs":{"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972580,"kb_used_data":20972580,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456204,"statfs":{"total":53687074816,"available":32211152896,"internally_reserved":0,"allocated":21475921920,"data_stored":21475921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971820,"kb_used_data":20971820,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456964,"statfs":{"total":53687074816,"available":32211931136,"internally_reserved":0,"allocated":21475143680,"data_stored":21475143680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32211152896,"internally_reserved":0,"allocated":21475921920,"data_stored":21475921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53687074816,"available":32211931136,"internally_reserved":0,"allocated":21475143680,"data_stored":21475143680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:12:21.324 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-26T03:12:21.325 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-26T03:12:21.325 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-26T03:12:21.325 DEBUG:teuthology.orchestra.run.trial053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-26T03:12:21.402 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:21.398+0000 7fa03736f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:21.403 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:12:21.401+0000 7fa03736f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:12:21.542 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:12:21.542 INFO:teuthology.orchestra.run.trial053.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-26T03:12:21.551 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-26T03:12:21.551 INFO:teuthology.run_tasks:Running task ssh_keys... 2026-02-26T03:12:21.615 DEBUG:teuthology.orchestra.run.trial053:> echo 'StrictHostKeyChecking no 2026-02-26T03:12:21.615 DEBUG:teuthology.orchestra.run.trial053:> ' > /home/ubuntu/.ssh/config && echo 'UserKnownHostsFile ' /dev/null >> /home/ubuntu/.ssh/config && chmod 600 /home/ubuntu/.ssh/config 2026-02-26T03:12:21.631 INFO:teuthology.task.ssh_keys:generated public key AAAAB3NzaC1yc2EAAAADAQABAAABAQCpmIYV4CIfjrIJv/0o2sGPqoeUAcV7YMD7p7VNJBLsfgQUvonts63GA+ukMf09uHklBl6shfJOfgODxI8jou6/OLpjNm2iOmOVsGWt9hqNalkCPgBmQiAIcr/lkZZRgyAz0zHRFj17ABIjLnxnu0A+j2DaJ82GsG/Hlin0iguSsF+39G2KrVjCC+2vnYOYPDE3YzEVWbBnqypDBBsHJSGuaZozdDVcUWBVnmKqsnxTMZ8bL/0WZGucHkysSgpdnZrxA1LL5RKLtN2hmNBPiQVIajS0eS7iOOqaGKV+itkQQRUDCRuFmcQ6z8Hz4M6UKzfzhx70u/MOax81FgBynQpx 2026-02-26T03:12:21.631 INFO:teuthology.task.ssh_keys:pushing keys to trial053.front.sepia.ceph.com for ubuntu 2026-02-26T03:12:21.631 DEBUG:teuthology.orchestra.run.trial053:> rm -f -- /home/ubuntu/.ssh/id_rsa 2026-02-26T03:12:21.686 DEBUG:teuthology.orchestra.run.trial053:> set -ex 2026-02-26T03:12:21.687 DEBUG:teuthology.orchestra.run.trial053:> dd of=/home/ubuntu/.ssh/id_rsa 2026-02-26T03:12:21.687 DEBUG:teuthology.orchestra.run.trial053:> chmod 0500 /home/ubuntu/.ssh/id_rsa 2026-02-26T03:12:21.744 DEBUG:teuthology.orchestra.run.trial053:> rm -f -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-26T03:12:21.801 DEBUG:teuthology.orchestra.run.trial053:> set -ex 2026-02-26T03:12:21.801 DEBUG:teuthology.orchestra.run.trial053:> dd of=/home/ubuntu/.ssh/id_rsa.pub 2026-02-26T03:12:21.857 DEBUG:teuthology.orchestra.run.trial053:> cp -v -a /home/ubuntu/.ssh/authorized_keys /home/ubuntu/.ssh/authorized_keys_2026-02-26_03:12:21:856986 2026-02-26T03:12:21.914 INFO:teuthology.orchestra.run.trial053.stdout:'/home/ubuntu/.ssh/authorized_keys' -> '/home/ubuntu/.ssh/authorized_keys_2026-02-26_03:12:21:856986' 2026-02-26T03:12:21.914 DEBUG:teuthology.orchestra.run.trial053:> set -ex 2026-02-26T03:12:21.914 DEBUG:teuthology.orchestra.run.trial053:> dd of=/home/ubuntu/.ssh/authorized_keys conv=notrunc oflag=append 2026-02-26T03:12:21.972 INFO:teuthology.run_tasks:Running task cbt... 2026-02-26T03:12:21.998 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@trial053.front.sepia.ceph.com', 'osds': ['ubuntu@trial053.front.sepia.ceph.com'], 'mons': ['ubuntu@trial053.front.sepia.ceph.com'], 'clients': ['ubuntu@trial053.front.sepia.ceph.com'], 'rgws': {'ubuntu@trial053.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randrw'], 'norandommap': True, 'op_size': [4096], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2], 'cmd_path': '/home/ubuntu/cephtest/fio/fio'}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2026-02-26T03:12:21.998 DEBUG:teuthology.orchestra.run.trial053:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2026-02-26T03:12:22.033 DEBUG:teuthology.orchestra.run.trial053:> set -ex 2026-02-26T03:12:22.033 DEBUG:teuthology.orchestra.run.trial053:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2026-02-26T03:12:22.085 DEBUG:teuthology.orchestra.run.trial053:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2026-02-26T03:12:22.147 INFO:teuthology.orchestra.run.trial053.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2026-02-26T03:12:22.378 INFO:tasks.cbt:Installing collectl 2026-02-26T03:12:22.378 DEBUG:teuthology.orchestra.run.trial053:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2026-02-26T03:12:22.453 INFO:teuthology.orchestra.run.trial053.stderr:--2026-02-26 03:12:22-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2026-02-26T03:12:22.537 INFO:teuthology.orchestra.run.trial053.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.12.149, 104.18.13.149, 2606:4700::6812:d95, ... 2026-02-26T03:12:22.540 INFO:teuthology.orchestra.run.trial053.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.12.149|:443... connected. 2026-02-26T03:12:22.732 INFO:teuthology.orchestra.run.trial053.stderr:HTTP request sent, awaiting response... 302 Found 2026-02-26T03:12:22.733 INFO:teuthology.orchestra.run.trial053.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABpn7oWi0ieH2fPPCwrRXW8aSqgdGLGmdYOexDS3ERwGrto-TFHpAjxjL3hGoSsVJk6un1GgodcbmcR5xmUdBVrRwPQ0g%3D%3D&use_mirror=phoenixnap&r= [following] 2026-02-26T03:12:22.733 INFO:teuthology.orchestra.run.trial053.stderr:--2026-02-26 03:12:22-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABpn7oWi0ieH2fPPCwrRXW8aSqgdGLGmdYOexDS3ERwGrto-TFHpAjxjL3hGoSsVJk6un1GgodcbmcR5xmUdBVrRwPQ0g%3D%3D&use_mirror=phoenixnap&r= 2026-02-26T03:12:22.770 INFO:teuthology.orchestra.run.trial053.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 104.18.12.149, 104.18.13.149, 2606:4700::6812:d95, ... 2026-02-26T03:12:22.773 INFO:teuthology.orchestra.run.trial053.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|104.18.12.149|:443... connected. 2026-02-26T03:12:23.054 INFO:teuthology.orchestra.run.trial053.stderr:HTTP request sent, awaiting response... 302 Found 2026-02-26T03:12:23.054 INFO:teuthology.orchestra.run.trial053.stderr:Location: https://phoenixnap.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 [following] 2026-02-26T03:12:23.054 INFO:teuthology.orchestra.run.trial053.stderr:--2026-02-26 03:12:23-- https://phoenixnap.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 2026-02-26T03:12:23.060 INFO:teuthology.orchestra.run.trial053.stderr:Resolving phoenixnap.dl.sourceforge.net (phoenixnap.dl.sourceforge.net)... 184.164.141.26 2026-02-26T03:12:23.127 INFO:teuthology.orchestra.run.trial053.stderr:Connecting to phoenixnap.dl.sourceforge.net (phoenixnap.dl.sourceforge.net)|184.164.141.26|:443... connected. 2026-02-26T03:12:23.534 INFO:teuthology.orchestra.run.trial053.stderr:HTTP request sent, awaiting response... 200 OK 2026-02-26T03:12:23.534 INFO:teuthology.orchestra.run.trial053.stderr:Length: 620693 (606K) [application/x-gzip] 2026-02-26T03:12:23.534 INFO:teuthology.orchestra.run.trial053.stderr:Saving to: ‘collectl.tar.gz’ 2026-02-26T03:12:23.534 INFO:teuthology.orchestra.run.trial053.stderr: 2026-02-26T03:12:26.120 INFO:teuthology.orchestra.run.trial053.stderr: 0K .......... .......... .......... .......... .......... 8% 19.3K 29s 2026-02-26T03:12:26.189 INFO:teuthology.orchestra.run.trial053.stderr: 50K .......... .......... .......... .......... .......... 16% 743K 13s 2026-02-26T03:12:26.255 INFO:teuthology.orchestra.run.trial053.stderr: 100K .......... .......... .......... .......... .......... 24% 746K 8s 2026-02-26T03:12:26.255 INFO:teuthology.orchestra.run.trial053.stderr: 150K .......... .......... .......... .......... .......... 32% 94.6M 6s 2026-02-26T03:12:26.324 INFO:teuthology.orchestra.run.trial053.stderr: 200K .......... .......... .......... .......... .......... 41% 749K 4s 2026-02-26T03:12:26.324 INFO:teuthology.orchestra.run.trial053.stderr: 250K .......... .......... .......... .......... .......... 49% 97.2M 3s 2026-02-26T03:12:26.325 INFO:teuthology.orchestra.run.trial053.stderr: 300K .......... .......... .......... .......... .......... 57% 171M 2s 2026-02-26T03:12:26.399 INFO:teuthology.orchestra.run.trial053.stderr: 350K .......... .......... .......... .......... .......... 65% 760K 1s 2026-02-26T03:12:26.400 INFO:teuthology.orchestra.run.trial053.stderr: 400K .......... .......... .......... .......... .......... 74% 131M 1s 2026-02-26T03:12:26.400 INFO:teuthology.orchestra.run.trial053.stderr: 450K .......... .......... .......... .......... .......... 82% 158M 1s 2026-02-26T03:12:26.400 INFO:teuthology.orchestra.run.trial053.stderr: 500K .......... .......... .......... .......... .......... 90% 235M 0s 2026-02-26T03:12:26.400 INFO:teuthology.orchestra.run.trial053.stderr: 550K .......... .......... .......... .......... .......... 98% 309M 0s 2026-02-26T03:12:26.400 INFO:teuthology.orchestra.run.trial053.stderr: 600K ...... 100% 11.4T=2.9s 2026-02-26T03:12:26.401 INFO:teuthology.orchestra.run.trial053.stderr: 2026-02-26T03:12:26.401 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26 03:12:26 (212 KB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2026-02-26T03:12:26.401 INFO:teuthology.orchestra.run.trial053.stderr: 2026-02-26T03:12:26.405 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/ 2026-02-26T03:12:26.405 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/collectl.conf 2026-02-26T03:12:26.405 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/ 2026-02-26T03:12:26.405 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/SlowProc.html 2026-02-26T03:12:26.405 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/BuddyInfo.html 2026-02-26T03:12:26.406 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/WhySummary.html 2026-02-26T03:12:26.406 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Socket.html 2026-02-26T03:12:26.406 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/colmux.html 2026-02-26T03:12:26.406 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Tutorial.html 2026-02-26T03:12:26.406 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Gexpr.html 2026-02-26T03:12:26.406 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Import.html 2026-02-26T03:12:26.406 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/style.css 2026-02-26T03:12:26.406 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2026-02-26T03:12:26.406 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2026-02-26T03:12:26.406 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/FileNaming.html 2026-02-26T03:12:26.406 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/TheMath.html 2026-02-26T03:12:26.406 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Environmental.html 2026-02-26T03:12:26.406 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Process.html 2026-02-26T03:12:26.406 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/OutputFiles.html 2026-02-26T03:12:26.406 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/HiResTime.html 2026-02-26T03:12:26.406 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/OutputFormats.html 2026-02-26T03:12:26.406 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/slub.jpg 2026-02-26T03:12:26.406 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Lustre.html 2026-02-26T03:12:26.406 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Documentation.html 2026-02-26T03:12:26.406 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Inodes.html 2026-02-26T03:12:26.406 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Graphite.html 2026-02-26T03:12:26.406 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/InputFiles.html 2026-02-26T03:12:26.406 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Export.html 2026-02-26T03:12:26.406 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Interrupts.html 2026-02-26T03:12:26.406 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Startup.html 2026-02-26T03:12:26.406 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Performance.html 2026-02-26T03:12:26.406 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Messages.html 2026-02-26T03:12:26.406 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Logging.html 2026-02-26T03:12:26.406 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Data-detail.html 2026-02-26T03:12:26.407 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/CPUs.html 2026-02-26T03:12:26.407 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Matrix.html 2026-02-26T03:12:26.407 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Data-brief.html 2026-02-26T03:12:26.407 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/SlabInfo.html 2026-02-26T03:12:26.407 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Misc.html 2026-02-26T03:12:26.407 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Examples.html 2026-02-26T03:12:26.407 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/OpenStack.html 2026-02-26T03:12:26.407 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Plotfiles.html 2026-02-26T03:12:26.407 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2026-02-26T03:12:26.407 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Data-verbose.html 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Data.html 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Exceptions.html 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/RunningAsAService.html 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Ganglia.jpg 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/OperationalModes.html 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Playback.html 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Hello.html 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Sockets.html 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Network.html 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Infiniband.html 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Memory.html 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/Disks.html 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/docs/NfsInfo.html 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/service/ 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/service/collectl.service 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/initd/ 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/initd/collectl-suse 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/initd/collectl-debian 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/initd/collectl-generic 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/initd/collectl 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/misc.ph 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/gexpr.ph 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/statsd.ph 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/envrules.std 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/hello.ph 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/vmsum.ph 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/proctree.ph 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/README-WINDOWS 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/formatit.ph 2026-02-26T03:12:26.408 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/colmux 2026-02-26T03:12:26.409 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/INSTALL 2026-02-26T03:12:26.409 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/graphite.ph 2026-02-26T03:12:26.409 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/RELEASE-collectl 2026-02-26T03:12:26.409 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/vmstat.ph 2026-02-26T03:12:26.409 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/README 2026-02-26T03:12:26.409 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/man1/ 2026-02-26T03:12:26.409 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/man1/collectl.1 2026-02-26T03:12:26.410 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/man1/colmux.1 2026-02-26T03:12:26.410 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/lexpr.ph 2026-02-26T03:12:26.410 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/client.pl 2026-02-26T03:12:26.410 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/collectl 2026-02-26T03:12:26.411 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/COPYING 2026-02-26T03:12:26.411 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/vnet.ph 2026-02-26T03:12:26.411 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/GPL 2026-02-26T03:12:26.411 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/UNINSTALL 2026-02-26T03:12:26.411 INFO:teuthology.orchestra.run.trial053.stdout:collectl-4.3.1/ARTISTIC 2026-02-26T03:12:26.455 INFO:teuthology.orchestra.run.trial053.stderr:./INSTALL: line 90: chkconfig: command not found 2026-02-26T03:12:26.456 DEBUG:teuthology.orchestra.run.trial053:> sudo yum -y install librbd-devel pdsh pdsh-rcmd-ssh perf 2026-02-26T03:12:26.822 INFO:teuthology.orchestra.run.trial053.stdout:Last metadata expiration check: 0:01:04 ago on Thu 26 Feb 2026 03:11:22 AM UTC. 2026-02-26T03:12:26.909 INFO:teuthology.orchestra.run.trial053.stdout:Dependencies resolved. 2026-02-26T03:12:26.909 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:12:26.909 INFO:teuthology.orchestra.run.trial053.stdout: Package Arch Version Repository Size 2026-02-26T03:12:26.909 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:12:26.909 INFO:teuthology.orchestra.run.trial053.stdout:Installing: 2026-02-26T03:12:26.909 INFO:teuthology.orchestra.run.trial053.stdout: librbd-devel x86_64 2:20.3.0-5312.gabffb617.el9 ceph 24 k 2026-02-26T03:12:26.909 INFO:teuthology.orchestra.run.trial053.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2026-02-26T03:12:26.910 INFO:teuthology.orchestra.run.trial053.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2026-02-26T03:12:26.910 INFO:teuthology.orchestra.run.trial053.stdout: perf x86_64 5.14.0-686.el9 appstream 3.3 M 2026-02-26T03:12:26.910 INFO:teuthology.orchestra.run.trial053.stdout:Installing dependencies: 2026-02-26T03:12:26.910 INFO:teuthology.orchestra.run.trial053.stdout: libradospp-devel x86_64 2:20.3.0-5312.gabffb617.el9 ceph 31 k 2026-02-26T03:12:26.910 INFO:teuthology.orchestra.run.trial053.stdout: libtraceevent x86_64 1.8.4-2.el9 baseos 290 k 2026-02-26T03:12:26.910 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:12:26.910 INFO:teuthology.orchestra.run.trial053.stdout:Transaction Summary 2026-02-26T03:12:26.910 INFO:teuthology.orchestra.run.trial053.stdout:================================================================================ 2026-02-26T03:12:26.910 INFO:teuthology.orchestra.run.trial053.stdout:Install 6 Packages 2026-02-26T03:12:26.910 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:12:26.911 INFO:teuthology.orchestra.run.trial053.stdout:Total download size: 3.8 M 2026-02-26T03:12:26.911 INFO:teuthology.orchestra.run.trial053.stdout:Installed size: 14 M 2026-02-26T03:12:26.911 INFO:teuthology.orchestra.run.trial053.stdout:Downloading Packages: 2026-02-26T03:12:27.435 INFO:teuthology.orchestra.run.trial053.stdout:(1/6): libtraceevent-1.8.4-2.el9.x86_64.rpm 11 MB/s | 290 kB 00:00 2026-02-26T03:12:27.753 INFO:teuthology.orchestra.run.trial053.stdout:(2/6): libradospp-devel-20.3.0-5312.gabffb617.e 90 kB/s | 31 kB 00:00 2026-02-26T03:12:27.756 INFO:teuthology.orchestra.run.trial053.stdout:(3/6): librbd-devel-20.3.0-5312.gabffb617.el9.x 68 kB/s | 24 kB 00:00 2026-02-26T03:12:27.768 INFO:teuthology.orchestra.run.trial053.stdout:(4/6): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 1.1 MB/s | 13 kB 00:00 2026-02-26T03:12:27.820 INFO:teuthology.orchestra.run.trial053.stdout:(5/6): pdsh-2.34-7.el9.x86_64.rpm 1.7 MB/s | 118 kB 00:00 2026-02-26T03:12:31.742 INFO:teuthology.orchestra.run.trial053.stdout:(6/6): perf-5.14.0-686.el9.x86_64.rpm 784 kB/s | 3.3 MB 00:04 2026-02-26T03:12:31.745 INFO:teuthology.orchestra.run.trial053.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:12:31.745 INFO:teuthology.orchestra.run.trial053.stdout:Total 797 kB/s | 3.8 MB 00:04 2026-02-26T03:12:31.769 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction check 2026-02-26T03:12:31.779 INFO:teuthology.orchestra.run.trial053.stdout:Transaction check succeeded. 2026-02-26T03:12:31.779 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction test 2026-02-26T03:12:31.837 INFO:teuthology.orchestra.run.trial053.stdout:Transaction test succeeded. 2026-02-26T03:12:31.837 INFO:teuthology.orchestra.run.trial053.stdout:Running transaction 2026-02-26T03:12:31.912 INFO:teuthology.orchestra.run.trial053.stdout: Preparing : 1/1 2026-02-26T03:12:31.917 INFO:teuthology.orchestra.run.trial053.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/6 2026-02-26T03:12:31.956 INFO:teuthology.orchestra.run.trial053.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/6 2026-02-26T03:12:31.960 INFO:teuthology.orchestra.run.trial053.stdout: Installing : libtraceevent-1.8.4-2.el9.x86_64 3/6 2026-02-26T03:12:31.963 INFO:teuthology.orchestra.run.trial053.stdout: Installing : libradospp-devel-2:20.3.0-5312.gabffb617.el9.x86_64 4/6 2026-02-26T03:12:32.128 INFO:teuthology.orchestra.run.trial053.stdout: Installing : librbd-devel-2:20.3.0-5312.gabffb617.el9.x86_64 5/6 2026-02-26T03:12:32.150 INFO:teuthology.orchestra.run.trial053.stdout: Installing : perf-5.14.0-686.el9.x86_64 6/6 2026-02-26T03:12:32.263 INFO:teuthology.orchestra.run.trial053.stdout: Running scriptlet: perf-5.14.0-686.el9.x86_64 6/6 2026-02-26T03:12:32.264 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : libradospp-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/6 2026-02-26T03:12:32.264 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : librbd-devel-2:20.3.0-5312.gabffb617.el9.x86_64 2/6 2026-02-26T03:12:32.264 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : libtraceevent-1.8.4-2.el9.x86_64 3/6 2026-02-26T03:12:32.264 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : perf-5.14.0-686.el9.x86_64 4/6 2026-02-26T03:12:32.264 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : pdsh-2.34-7.el9.x86_64 5/6 2026-02-26T03:12:32.328 INFO:teuthology.orchestra.run.trial053.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 6/6 2026-02-26T03:12:32.328 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:12:32.328 INFO:teuthology.orchestra.run.trial053.stdout:Installed: 2026-02-26T03:12:32.328 INFO:teuthology.orchestra.run.trial053.stdout: libradospp-devel-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:12:32.329 INFO:teuthology.orchestra.run.trial053.stdout: librbd-devel-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:12:32.329 INFO:teuthology.orchestra.run.trial053.stdout: libtraceevent-1.8.4-2.el9.x86_64 2026-02-26T03:12:32.329 INFO:teuthology.orchestra.run.trial053.stdout: pdsh-2.34-7.el9.x86_64 2026-02-26T03:12:32.329 INFO:teuthology.orchestra.run.trial053.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2026-02-26T03:12:32.329 INFO:teuthology.orchestra.run.trial053.stdout: perf-5.14.0-686.el9.x86_64 2026-02-26T03:12:32.329 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:12:32.329 INFO:teuthology.orchestra.run.trial053.stdout:Complete! 2026-02-26T03:12:32.359 DEBUG:teuthology.orchestra.run.trial053:> python3 -m pip install -r /home/ubuntu/cephtest/cbt/requirements.txt 2026-02-26T03:12:32.497 INFO:teuthology.orchestra.run.trial053.stdout:Defaulting to user installation because normal site-packages is not writeable 2026-02-26T03:12:32.509 INFO:teuthology.orchestra.run.trial053.stdout:Requirement already satisfied: pyyaml in /usr/lib64/python3.9/site-packages (from -r /home/ubuntu/cephtest/cbt/requirements.txt (line 1)) (5.4.1) 2026-02-26T03:12:32.509 INFO:teuthology.orchestra.run.trial053.stdout:Requirement already satisfied: lxml in /usr/lib64/python3.9/site-packages (from -r /home/ubuntu/cephtest/cbt/requirements.txt (line 2)) (4.6.5) 2026-02-26T03:12:32.841 INFO:teuthology.orchestra.run.trial053.stdout:Collecting matplotlib 2026-02-26T03:12:32.874 INFO:teuthology.orchestra.run.trial053.stdout: Downloading matplotlib-3.9.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.3 MB) 2026-02-26T03:12:33.125 INFO:teuthology.orchestra.run.trial053.stdout:Collecting contourpy>=1.0.1 2026-02-26T03:12:33.130 INFO:teuthology.orchestra.run.trial053.stdout: Downloading contourpy-1.3.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (321 kB) 2026-02-26T03:12:33.143 INFO:teuthology.orchestra.run.trial053.stdout:Collecting cycler>=0.10 2026-02-26T03:12:33.147 INFO:teuthology.orchestra.run.trial053.stdout: Downloading cycler-0.12.1-py3-none-any.whl (8.3 kB) 2026-02-26T03:12:33.550 INFO:teuthology.orchestra.run.trial053.stdout:Collecting pillow>=8 2026-02-26T03:12:33.555 INFO:teuthology.orchestra.run.trial053.stdout: Downloading pillow-11.3.0-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (6.6 MB) 2026-02-26T03:12:33.625 INFO:teuthology.orchestra.run.trial053.stdout:Requirement already satisfied: numpy>=1.23 in /usr/lib64/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (1.23.5) 2026-02-26T03:12:33.625 INFO:teuthology.orchestra.run.trial053.stdout:Requirement already satisfied: pyparsing>=2.3.1 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (2.4.7) 2026-02-26T03:12:33.625 INFO:teuthology.orchestra.run.trial053.stdout:Requirement already satisfied: packaging>=20.0 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (20.9) 2026-02-26T03:12:33.951 INFO:teuthology.orchestra.run.trial053.stdout:Collecting fonttools>=4.22.0 2026-02-26T03:12:33.956 INFO:teuthology.orchestra.run.trial053.stdout: Downloading fonttools-4.60.2-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (4.8 MB) 2026-02-26T03:12:33.986 INFO:teuthology.orchestra.run.trial053.stdout:Requirement already satisfied: python-dateutil>=2.7 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (2.9.0.post0) 2026-02-26T03:12:34.150 INFO:teuthology.orchestra.run.trial053.stdout:Collecting kiwisolver>=1.3.1 2026-02-26T03:12:34.159 INFO:teuthology.orchestra.run.trial053.stdout: Downloading kiwisolver-1.4.7-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB) 2026-02-26T03:12:34.210 INFO:teuthology.orchestra.run.trial053.stdout:Collecting importlib-resources>=3.2.0 2026-02-26T03:12:34.214 INFO:teuthology.orchestra.run.trial053.stdout: Downloading importlib_resources-6.5.2-py3-none-any.whl (37 kB) 2026-02-26T03:12:34.264 INFO:teuthology.orchestra.run.trial053.stdout:Collecting zipp>=3.1.0 2026-02-26T03:12:34.268 INFO:teuthology.orchestra.run.trial053.stdout: Downloading zipp-3.23.0-py3-none-any.whl (10 kB) 2026-02-26T03:12:34.279 INFO:teuthology.orchestra.run.trial053.stdout:Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil>=2.7->matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (1.15.0) 2026-02-26T03:12:34.393 INFO:teuthology.orchestra.run.trial053.stdout:Installing collected packages: zipp, pillow, kiwisolver, importlib-resources, fonttools, cycler, contourpy, matplotlib 2026-02-26T03:12:35.459 INFO:teuthology.orchestra.run.trial053.stdout:Successfully installed contourpy-1.3.0 cycler-0.12.1 fonttools-4.60.2 importlib-resources-6.5.2 kiwisolver-1.4.7 matplotlib-3.9.4 pillow-11.3.0 zipp-3.23.0 2026-02-26T03:12:35.568 INFO:tasks.cbt:benchmark: librbdfio 2026-02-26T03:12:35.568 DEBUG:teuthology.orchestra.run.trial053:> git clone -b master https://github.com/axboe/fio.git /home/ubuntu/cephtest/fio 2026-02-26T03:12:35.585 INFO:teuthology.orchestra.run.trial053.stderr:Cloning into '/home/ubuntu/cephtest/fio'... 2026-02-26T03:12:36.376 DEBUG:teuthology.orchestra.run.trial053:> cd /home/ubuntu/cephtest/fio && ./configure && make 2026-02-26T03:12:36.545 INFO:teuthology.orchestra.run.trial053.stdout:Operating system Linux 2026-02-26T03:12:36.546 INFO:teuthology.orchestra.run.trial053.stdout:CPU x86_64 2026-02-26T03:12:36.546 INFO:teuthology.orchestra.run.trial053.stdout:Big endian no 2026-02-26T03:12:36.546 INFO:teuthology.orchestra.run.trial053.stdout:Compiler gcc 2026-02-26T03:12:36.546 INFO:teuthology.orchestra.run.trial053.stdout:Cross compile no 2026-02-26T03:12:36.546 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:12:36.546 INFO:teuthology.orchestra.run.trial053.stdout:Static build no 2026-02-26T03:12:36.591 INFO:teuthology.orchestra.run.trial053.stdout:Wordsize 64 2026-02-26T03:12:36.597 INFO:teuthology.orchestra.run.trial053.stdout:zlib no 2026-02-26T03:12:36.608 INFO:teuthology.orchestra.run.trial053.stdout:fcntl(F_FULLFSYNC) no 2026-02-26T03:12:36.618 INFO:teuthology.orchestra.run.trial053.stdout:Linux AIO support no 2026-02-26T03:12:36.619 INFO:teuthology.orchestra.run.trial053.stdout:Linux AIO support rw flags no 2026-02-26T03:12:36.642 INFO:teuthology.orchestra.run.trial053.stdout:POSIX AIO support yes 2026-02-26T03:12:36.642 INFO:teuthology.orchestra.run.trial053.stdout:POSIX AIO support needs -lrt no 2026-02-26T03:12:36.663 INFO:teuthology.orchestra.run.trial053.stdout:POSIX AIO fsync yes 2026-02-26T03:12:36.683 INFO:teuthology.orchestra.run.trial053.stdout:POSIX pshared support yes 2026-02-26T03:12:36.710 INFO:teuthology.orchestra.run.trial053.stdout:pthread_condattr_setclock() yes 2026-02-26T03:12:36.734 INFO:teuthology.orchestra.run.trial053.stdout:pthread_sigmask() yes 2026-02-26T03:12:36.757 INFO:teuthology.orchestra.run.trial053.stdout:pthread_getaffinity_np() yes 2026-02-26T03:12:36.763 INFO:teuthology.orchestra.run.trial053.stdout:Solaris AIO support no 2026-02-26T03:12:36.785 INFO:teuthology.orchestra.run.trial053.stdout:__sync_fetch_and_add yes 2026-02-26T03:12:36.806 INFO:teuthology.orchestra.run.trial053.stdout:__sync_synchronize yes 2026-02-26T03:12:36.826 INFO:teuthology.orchestra.run.trial053.stdout:__sync_val_compare_and_swap yes 2026-02-26T03:12:36.831 INFO:teuthology.orchestra.run.trial053.stdout:libverbs no 2026-02-26T03:12:36.840 INFO:teuthology.orchestra.run.trial053.stdout:rdmacm no 2026-02-26T03:12:36.861 INFO:teuthology.orchestra.run.trial053.stdout:asprintf() yes 2026-02-26T03:12:36.882 INFO:teuthology.orchestra.run.trial053.stdout:vasprintf() yes 2026-02-26T03:12:36.903 INFO:teuthology.orchestra.run.trial053.stdout:Linux fallocate yes 2026-02-26T03:12:36.926 INFO:teuthology.orchestra.run.trial053.stdout:POSIX fadvise yes 2026-02-26T03:12:36.946 INFO:teuthology.orchestra.run.trial053.stdout:POSIX fallocate yes 2026-02-26T03:12:36.966 INFO:teuthology.orchestra.run.trial053.stdout:sched_setaffinity(3 arg) yes 2026-02-26T03:12:36.966 INFO:teuthology.orchestra.run.trial053.stdout:sched_setaffinity(2 arg) no 2026-02-26T03:12:36.986 INFO:teuthology.orchestra.run.trial053.stdout:clock_gettime yes 2026-02-26T03:12:37.007 INFO:teuthology.orchestra.run.trial053.stdout:CLOCK_MONOTONIC yes 2026-02-26T03:12:37.029 INFO:teuthology.orchestra.run.trial053.stdout:gettimeofday yes 2026-02-26T03:12:37.051 INFO:teuthology.orchestra.run.trial053.stdout:fdatasync yes 2026-02-26T03:12:37.072 INFO:teuthology.orchestra.run.trial053.stdout:pipe() yes 2026-02-26T03:12:37.094 INFO:teuthology.orchestra.run.trial053.stdout:pipe2() yes 2026-02-26T03:12:37.115 INFO:teuthology.orchestra.run.trial053.stdout:pread() yes 2026-02-26T03:12:37.139 INFO:teuthology.orchestra.run.trial053.stdout:sync_file_range yes 2026-02-26T03:12:37.144 INFO:teuthology.orchestra.run.trial053.stdout:ASharedMemory_create no 2026-02-26T03:12:37.152 INFO:teuthology.orchestra.run.trial053.stdout:EXT4 move extent yes 2026-02-26T03:12:37.174 INFO:teuthology.orchestra.run.trial053.stdout:Linux splice(2) yes 2026-02-26T03:12:37.180 INFO:teuthology.orchestra.run.trial053.stdout:libnuma no 2026-02-26T03:12:37.199 INFO:teuthology.orchestra.run.trial053.stdout:strsep yes 2026-02-26T03:12:37.219 INFO:teuthology.orchestra.run.trial053.stdout:strcasestr yes 2026-02-26T03:12:37.228 INFO:teuthology.orchestra.run.trial053.stdout:strlcat no 2026-02-26T03:12:37.250 INFO:teuthology.orchestra.run.trial053.stdout:getopt_long_only() yes 2026-02-26T03:12:37.274 INFO:teuthology.orchestra.run.trial053.stdout:inet_aton yes 2026-02-26T03:12:37.296 INFO:teuthology.orchestra.run.trial053.stdout:socklen_t yes 2026-02-26T03:12:37.317 INFO:teuthology.orchestra.run.trial053.stdout:__thread yes 2026-02-26T03:12:37.338 INFO:teuthology.orchestra.run.trial053.stdout:RUSAGE_THREAD yes 2026-02-26T03:12:37.359 INFO:teuthology.orchestra.run.trial053.stdout:SCHED_IDLE yes 2026-02-26T03:12:37.382 INFO:teuthology.orchestra.run.trial053.stdout:TCP_NODELAY yes 2026-02-26T03:12:37.405 INFO:teuthology.orchestra.run.trial053.stdout:vsock yes 2026-02-26T03:12:37.427 INFO:teuthology.orchestra.run.trial053.stdout:Net engine window_size yes 2026-02-26T03:12:37.449 INFO:teuthology.orchestra.run.trial053.stdout:TCP_MAXSEG yes 2026-02-26T03:12:37.471 INFO:teuthology.orchestra.run.trial053.stdout:RLIMIT_MEMLOCK yes 2026-02-26T03:12:37.494 INFO:teuthology.orchestra.run.trial053.stdout:pwritev/preadv yes 2026-02-26T03:12:37.515 INFO:teuthology.orchestra.run.trial053.stdout:pwritev2/preadv2 yes 2026-02-26T03:12:37.539 INFO:teuthology.orchestra.run.trial053.stdout:IPv6 helpers yes 2026-02-26T03:12:37.550 INFO:teuthology.orchestra.run.trial053.stdout:http engine no 2026-02-26T03:12:37.601 INFO:teuthology.orchestra.run.trial053.stdout:Rados engine yes 2026-02-26T03:12:37.666 INFO:teuthology.orchestra.run.trial053.stdout:Rados Block Device engine yes 2026-02-26T03:12:37.729 INFO:teuthology.orchestra.run.trial053.stdout:rbd_poll yes 2026-02-26T03:12:37.792 INFO:teuthology.orchestra.run.trial053.stdout:rbd_invalidate_cache yes 2026-02-26T03:12:37.859 INFO:teuthology.orchestra.run.trial053.stdout:rbd_encryption_load yes 2026-02-26T03:12:37.880 INFO:teuthology.orchestra.run.trial053.stdout:setvbuf yes 2026-02-26T03:12:37.885 INFO:teuthology.orchestra.run.trial053.stdout:Gluster API engine no 2026-02-26T03:12:37.895 INFO:teuthology.orchestra.run.trial053.stdout:s390_z196_facilities no 2026-02-26T03:12:37.895 INFO:teuthology.orchestra.run.trial053.stdout:HDFS engine no 2026-02-26T03:12:37.917 INFO:teuthology.orchestra.run.trial053.stdout:MTD yes 2026-02-26T03:12:37.923 INFO:teuthology.orchestra.run.trial053.stdout:libpmem no 2026-02-26T03:12:37.923 INFO:teuthology.orchestra.run.trial053.stdout:libpmem1_5 no 2026-02-26T03:12:37.928 INFO:teuthology.orchestra.run.trial053.stdout:libpmem2 no 2026-02-26T03:12:37.928 INFO:teuthology.orchestra.run.trial053.stdout:PMDK dev-dax engine no 2026-02-26T03:12:37.928 INFO:teuthology.orchestra.run.trial053.stdout:PMDK libpmem engine no 2026-02-26T03:12:37.933 INFO:teuthology.orchestra.run.trial053.stdout:DDN's Infinite Memory Engine no 2026-02-26T03:12:37.933 INFO:teuthology.orchestra.run.trial053.stdout:iscsi engine no 2026-02-26T03:12:37.933 INFO:teuthology.orchestra.run.trial053.stdout:NBD engine no 2026-02-26T03:12:37.938 INFO:teuthology.orchestra.run.trial053.stdout:DAOS File System (dfs) Engine no 2026-02-26T03:12:37.945 INFO:teuthology.orchestra.run.trial053.stdout:NFS engine no 2026-02-26T03:12:37.945 INFO:teuthology.orchestra.run.trial053.stdout:lex/yacc for arithmetic no 2026-02-26T03:12:37.966 INFO:teuthology.orchestra.run.trial053.stdout:getmntent yes 2026-02-26T03:12:37.978 INFO:teuthology.orchestra.run.trial053.stdout:getmntinfo no 2026-02-26T03:12:38.009 INFO:teuthology.orchestra.run.trial053.stdout:Static Assert yes 2026-02-26T03:12:38.028 INFO:teuthology.orchestra.run.trial053.stdout:bool yes 2026-02-26T03:12:38.050 INFO:teuthology.orchestra.run.trial053.stdout:strndup yes 2026-02-26T03:12:38.056 INFO:teuthology.orchestra.run.trial053.stdout:Valgrind headers no 2026-02-26T03:12:38.075 INFO:teuthology.orchestra.run.trial053.stdout:Zoned block device support yes 2026-02-26T03:12:38.094 INFO:teuthology.orchestra.run.trial053.stdout:Zoned block device capacity yes 2026-02-26T03:12:38.100 INFO:teuthology.orchestra.run.trial053.stdout:libzbc engine no 2026-02-26T03:12:38.119 INFO:teuthology.orchestra.run.trial053.stdout:NVMe uring command support yes 2026-02-26T03:12:38.131 INFO:teuthology.orchestra.run.trial053.stdout:xnvme engine no 2026-02-26T03:12:38.137 INFO:teuthology.orchestra.run.trial053.stdout:isal no 2026-02-26T03:12:38.137 INFO:teuthology.orchestra.run.trial053.stdout:isal CRC64 no 2026-02-26T03:12:38.149 INFO:teuthology.orchestra.run.trial053.stdout:libblkio engine no 2026-02-26T03:12:38.149 INFO:teuthology.orchestra.run.trial053.stdout:march_armv8_a_crc_crypto no 2026-02-26T03:12:38.150 INFO:teuthology.orchestra.run.trial053.stdout:cuda no 2026-02-26T03:12:38.150 INFO:teuthology.orchestra.run.trial053.stdout:libcufile no 2026-02-26T03:12:38.170 INFO:teuthology.orchestra.run.trial053.stdout:Build march=native yes 2026-02-26T03:12:38.176 INFO:teuthology.orchestra.run.trial053.stdout:CUnit no 2026-02-26T03:12:38.195 INFO:teuthology.orchestra.run.trial053.stdout:__kernel_rwf_t yes 2026-02-26T03:12:38.214 INFO:teuthology.orchestra.run.trial053.stdout:-Wimplicit-fallthrough=2 yes 2026-02-26T03:12:38.235 INFO:teuthology.orchestra.run.trial053.stdout:-Wno-stringop-truncation yes 2026-02-26T03:12:38.255 INFO:teuthology.orchestra.run.trial053.stdout:MADV_HUGEPAGE yes 2026-02-26T03:12:38.275 INFO:teuthology.orchestra.run.trial053.stdout:gettid yes 2026-02-26T03:12:38.298 INFO:teuthology.orchestra.run.trial053.stdout:statx(2)/libc yes 2026-02-26T03:12:38.321 INFO:teuthology.orchestra.run.trial053.stdout:statx(2)/syscall yes 2026-02-26T03:12:38.323 INFO:teuthology.orchestra.run.trial053.stdout:Windows PDB generation no 2026-02-26T03:12:38.343 INFO:teuthology.orchestra.run.trial053.stdout:timerfd_create yes 2026-02-26T03:12:38.344 INFO:teuthology.orchestra.run.trial053.stdout:Consider installing zlib1g-dev (zlib-devel) as some fio features depend on it. 2026-02-26T03:12:38.345 INFO:teuthology.orchestra.run.trial053.stdout:Lib-based ioengines dynamic no 2026-02-26T03:12:38.388 INFO:teuthology.orchestra.run.trial053.stdout:TCMalloc support yes 2026-02-26T03:12:38.389 INFO:teuthology.orchestra.run.trial053.stdout:seed_buckets 4 2026-02-26T03:12:38.434 INFO:teuthology.orchestra.run.trial053.stderr:FIO_VERSION = fio-3.41-113-g411de 2026-02-26T03:12:38.457 INFO:teuthology.orchestra.run.trial053.stdout: CC crc/crc16.o 2026-02-26T03:12:38.493 INFO:teuthology.orchestra.run.trial053.stdout: CC crc/crc32.o 2026-02-26T03:12:38.521 INFO:teuthology.orchestra.run.trial053.stdout: CC crc/crc32c-arm64.o 2026-02-26T03:12:38.564 INFO:teuthology.orchestra.run.trial053.stdout: CC crc/crc32c-intel.o 2026-02-26T03:12:38.595 INFO:teuthology.orchestra.run.trial053.stdout: CC crc/crc32c.o 2026-02-26T03:12:38.625 INFO:teuthology.orchestra.run.trial053.stdout: CC crc/crc64.o 2026-02-26T03:12:38.653 INFO:teuthology.orchestra.run.trial053.stdout: CC crc/crc7.o 2026-02-26T03:12:38.679 INFO:teuthology.orchestra.run.trial053.stdout: CC crc/crct10dif_common.o 2026-02-26T03:12:38.704 INFO:teuthology.orchestra.run.trial053.stdout: CC crc/fnv.o 2026-02-26T03:12:38.733 INFO:teuthology.orchestra.run.trial053.stdout: CC crc/md5.o 2026-02-26T03:12:38.790 INFO:teuthology.orchestra.run.trial053.stdout: CC crc/murmur3.o 2026-02-26T03:12:38.821 INFO:teuthology.orchestra.run.trial053.stdout: CC crc/sha1.o 2026-02-26T03:12:38.920 INFO:teuthology.orchestra.run.trial053.stdout: CC crc/sha256.o 2026-02-26T03:12:39.060 INFO:teuthology.orchestra.run.trial053.stdout: CC crc/sha3.o 2026-02-26T03:12:39.141 INFO:teuthology.orchestra.run.trial053.stdout: CC crc/sha512.o 2026-02-26T03:12:39.208 INFO:teuthology.orchestra.run.trial053.stdout: CC crc/test.o 2026-02-26T03:12:39.299 INFO:teuthology.orchestra.run.trial053.stdout: CC crc/xxhash.o 2026-02-26T03:12:39.357 INFO:teuthology.orchestra.run.trial053.stdout: CC lib/axmap.o 2026-02-26T03:12:39.429 INFO:teuthology.orchestra.run.trial053.stdout: CC lib/bloom.o 2026-02-26T03:12:39.480 INFO:teuthology.orchestra.run.trial053.stdout: CC lib/flist_sort.o 2026-02-26T03:12:39.526 INFO:teuthology.orchestra.run.trial053.stdout: CC lib/gauss.o 2026-02-26T03:12:39.577 INFO:teuthology.orchestra.run.trial053.stdout: CC lib/getrusage.o 2026-02-26T03:12:39.604 INFO:teuthology.orchestra.run.trial053.stdout: CC lib/hweight.o 2026-02-26T03:12:39.632 INFO:teuthology.orchestra.run.trial053.stdout: CC lib/ieee754.o 2026-02-26T03:12:39.664 INFO:teuthology.orchestra.run.trial053.stdout: CC lib/lfsr.o 2026-02-26T03:12:39.721 INFO:teuthology.orchestra.run.trial053.stdout: CC lib/memalign.o 2026-02-26T03:12:39.753 INFO:teuthology.orchestra.run.trial053.stdout: CC lib/memcpy.o 2026-02-26T03:12:39.835 INFO:teuthology.orchestra.run.trial053.stdout: CC lib/mountcheck.o 2026-02-26T03:12:39.866 INFO:teuthology.orchestra.run.trial053.stdout: CC lib/num2str.o 2026-02-26T03:12:39.910 INFO:teuthology.orchestra.run.trial053.stdout: CC lib/output_buffer.o 2026-02-26T03:12:39.945 INFO:teuthology.orchestra.run.trial053.stdout: CC lib/pattern.o 2026-02-26T03:12:40.026 INFO:teuthology.orchestra.run.trial053.stdout: CC lib/prio_tree.o 2026-02-26T03:12:40.096 INFO:teuthology.orchestra.run.trial053.stdout: CC lib/rand.o 2026-02-26T03:12:40.171 INFO:teuthology.orchestra.run.trial053.stdout: CC lib/rbtree.o 2026-02-26T03:12:40.241 INFO:teuthology.orchestra.run.trial053.stdout: CC lib/strntol.o 2026-02-26T03:12:40.276 INFO:teuthology.orchestra.run.trial053.stdout: CC lib/zipf.o 2026-02-26T03:12:40.344 INFO:teuthology.orchestra.run.trial053.stdout: CC gettime.o 2026-02-26T03:12:40.496 INFO:teuthology.orchestra.run.trial053.stdout: CC ioengines.o 2026-02-26T03:12:40.632 INFO:teuthology.orchestra.run.trial053.stdout: CC init.o 2026-02-26T03:12:41.010 INFO:teuthology.orchestra.run.trial053.stdout: CC stat.o 2026-02-26T03:12:41.722 INFO:teuthology.orchestra.run.trial053.stdout: CC log.o 2026-02-26T03:12:41.794 INFO:teuthology.orchestra.run.trial053.stdout: CC time.o 2026-02-26T03:12:41.885 INFO:teuthology.orchestra.run.trial053.stdout: CC filesetup.o 2026-02-26T03:12:42.220 INFO:teuthology.orchestra.run.trial053.stdout: CC eta.o 2026-02-26T03:12:42.383 INFO:teuthology.orchestra.run.trial053.stdout: CC verify.o 2026-02-26T03:12:42.721 INFO:teuthology.orchestra.run.trial053.stdout: CC memory.o 2026-02-26T03:12:42.818 INFO:teuthology.orchestra.run.trial053.stdout: CC io_u.o 2026-02-26T03:12:43.205 INFO:teuthology.orchestra.run.trial053.stdout: CC parse.o 2026-02-26T03:12:43.428 INFO:teuthology.orchestra.run.trial053.stdout: CC fio_sem.o 2026-02-26T03:12:43.493 INFO:teuthology.orchestra.run.trial053.stdout: CC rwlock.o 2026-02-26T03:12:43.545 INFO:teuthology.orchestra.run.trial053.stdout: CC pshared.o 2026-02-26T03:12:43.588 INFO:teuthology.orchestra.run.trial053.stdout: CC options.o 2026-02-26T03:12:43.992 INFO:teuthology.orchestra.run.trial053.stdout: CC fio_shared_sem.o 2026-02-26T03:12:44.025 INFO:teuthology.orchestra.run.trial053.stdout: CC smalloc.o 2026-02-26T03:12:44.127 INFO:teuthology.orchestra.run.trial053.stdout: CC filehash.o 2026-02-26T03:12:44.213 INFO:teuthology.orchestra.run.trial053.stdout: CC profile.o 2026-02-26T03:12:44.283 INFO:teuthology.orchestra.run.trial053.stdout: CC debug.o 2026-02-26T03:12:44.314 INFO:teuthology.orchestra.run.trial053.stdout: CC engines/cpu.o 2026-02-26T03:12:44.393 INFO:teuthology.orchestra.run.trial053.stdout: CC engines/mmap.o 2026-02-26T03:12:44.483 INFO:teuthology.orchestra.run.trial053.stdout: CC engines/sync.o 2026-02-26T03:12:44.605 INFO:teuthology.orchestra.run.trial053.stdout: CC engines/null.o 2026-02-26T03:12:44.676 INFO:teuthology.orchestra.run.trial053.stdout: CC engines/net.o 2026-02-26T03:12:44.882 INFO:teuthology.orchestra.run.trial053.stdout: CC engines/ftruncate.o 2026-02-26T03:12:44.944 INFO:teuthology.orchestra.run.trial053.stdout: CC engines/fileoperations.o 2026-02-26T03:12:45.031 INFO:teuthology.orchestra.run.trial053.stdout: CC engines/exec.o 2026-02-26T03:12:45.115 INFO:teuthology.orchestra.run.trial053.stdout: CC server.o 2026-02-26T03:12:45.471 INFO:teuthology.orchestra.run.trial053.stdout: CC client.o 2026-02-26T03:12:45.752 INFO:teuthology.orchestra.run.trial053.stdout: CC iolog.o 2026-02-26T03:12:45.980 INFO:teuthology.orchestra.run.trial053.stdout: CC backend.o 2026-02-26T03:12:46.396 INFO:teuthology.orchestra.run.trial053.stdout: CC libfio.o 2026-02-26T03:12:46.494 INFO:teuthology.orchestra.run.trial053.stdout: CC flow.o 2026-02-26T03:12:46.562 INFO:teuthology.orchestra.run.trial053.stdout: CC cconv.o 2026-02-26T03:12:46.784 INFO:teuthology.orchestra.run.trial053.stdout: CC gettime-thread.o 2026-02-26T03:12:46.847 INFO:teuthology.orchestra.run.trial053.stdout: CC helpers.o 2026-02-26T03:12:46.887 INFO:teuthology.orchestra.run.trial053.stdout: CC json.o 2026-02-26T03:12:46.985 INFO:teuthology.orchestra.run.trial053.stdout: CC idletime.o 2026-02-26T03:12:47.105 INFO:teuthology.orchestra.run.trial053.stdout: CC td_error.o 2026-02-26T03:12:47.167 INFO:teuthology.orchestra.run.trial053.stdout: CC profiles/tiobench.o 2026-02-26T03:12:47.226 INFO:teuthology.orchestra.run.trial053.stdout: CC profiles/act.o 2026-02-26T03:12:47.318 INFO:teuthology.orchestra.run.trial053.stdout: CC io_u_queue.o 2026-02-26T03:12:47.353 INFO:teuthology.orchestra.run.trial053.stdout: CC filelock.o 2026-02-26T03:12:47.426 INFO:teuthology.orchestra.run.trial053.stdout: CC workqueue.o 2026-02-26T03:12:47.522 INFO:teuthology.orchestra.run.trial053.stdout: CC rate-submit.o 2026-02-26T03:12:47.606 INFO:teuthology.orchestra.run.trial053.stdout: CC optgroup.o 2026-02-26T03:12:47.639 INFO:teuthology.orchestra.run.trial053.stdout: CC helper_thread.o 2026-02-26T03:12:47.740 INFO:teuthology.orchestra.run.trial053.stdout: CC steadystate.o 2026-02-26T03:12:47.885 INFO:teuthology.orchestra.run.trial053.stdout: CC zone-dist.o 2026-02-26T03:12:47.962 INFO:teuthology.orchestra.run.trial053.stdout: CC zbd.o 2026-02-26T03:12:48.327 INFO:teuthology.orchestra.run.trial053.stdout: CC dedupe.o 2026-02-26T03:12:48.408 INFO:teuthology.orchestra.run.trial053.stdout: CC dataplacement.o 2026-02-26T03:12:48.504 INFO:teuthology.orchestra.run.trial053.stdout: CC sprandom.o 2026-02-26T03:12:48.666 INFO:teuthology.orchestra.run.trial053.stdout: CC engines/posixaio.o 2026-02-26T03:12:48.744 INFO:teuthology.orchestra.run.trial053.stdout: CC engines/falloc.o 2026-02-26T03:12:48.812 INFO:teuthology.orchestra.run.trial053.stdout: CC engines/e4defrag.o 2026-02-26T03:12:48.889 INFO:teuthology.orchestra.run.trial053.stdout: CC engines/splice.o 2026-02-26T03:12:48.990 INFO:teuthology.orchestra.run.trial053.stdout: CC oslib/asprintf.o 2026-02-26T03:12:49.016 INFO:teuthology.orchestra.run.trial053.stdout: CC oslib/strlcat.o 2026-02-26T03:12:49.047 INFO:teuthology.orchestra.run.trial053.stdout: CC engines/mtd.o 2026-02-26T03:12:49.134 INFO:teuthology.orchestra.run.trial053.stdout: CC oslib/libmtd.o 2026-02-26T03:12:49.327 INFO:teuthology.orchestra.run.trial053.stdout: CC oslib/libmtd_legacy.o 2026-02-26T03:12:49.400 INFO:teuthology.orchestra.run.trial053.stdout: CC diskutil.o 2026-02-26T03:12:49.513 INFO:teuthology.orchestra.run.trial053.stdout: CC fifo.o 2026-02-26T03:12:49.551 INFO:teuthology.orchestra.run.trial053.stdout: CC blktrace.o 2026-02-26T03:12:49.719 INFO:teuthology.orchestra.run.trial053.stdout: CC cgroup.o 2026-02-26T03:12:49.803 INFO:teuthology.orchestra.run.trial053.stdout: CC trim.o 2026-02-26T03:12:49.871 INFO:teuthology.orchestra.run.trial053.stdout: CC engines/sg.o 2026-02-26T03:12:50.065 INFO:teuthology.orchestra.run.trial053.stdout: CC oslib/linux-dev-lookup.o 2026-02-26T03:12:50.103 INFO:teuthology.orchestra.run.trial053.stdout: CC engines/io_uring.o 2026-02-26T03:12:50.333 INFO:teuthology.orchestra.run.trial053.stdout: CC engines/nvme.o 2026-02-26T03:12:50.495 INFO:teuthology.orchestra.run.trial053.stdout: CC oslib/linux-blkzoned.o 2026-02-26T03:12:50.584 INFO:teuthology.orchestra.run.trial053.stdout: CC engines/cmdprio.o 2026-02-26T03:12:50.712 INFO:teuthology.orchestra.run.trial053.stdout: CC engines/rados.o 2026-02-26T03:12:50.814 INFO:teuthology.orchestra.run.trial053.stdout: CC engines/rbd.o 2026-02-26T03:12:50.919 INFO:teuthology.orchestra.run.trial053.stdout: CC fio.o 2026-02-26T03:12:50.974 INFO:teuthology.orchestra.run.trial053.stdout: LINK fio 2026-02-26T03:12:51.096 INFO:teuthology.orchestra.run.trial053.stdout: CC t/log.o 2026-02-26T03:12:51.129 INFO:teuthology.orchestra.run.trial053.stdout: CC t/genzipf.o 2026-02-26T03:12:51.204 INFO:teuthology.orchestra.run.trial053.stdout: CC oslib/strcasestr.o 2026-02-26T03:12:51.225 INFO:teuthology.orchestra.run.trial053.stdout: CC oslib/strndup.o 2026-02-26T03:12:51.244 INFO:teuthology.orchestra.run.trial053.stdout: LINK t/fio-genzipf 2026-02-26T03:12:51.301 INFO:teuthology.orchestra.run.trial053.stdout: CC t/btrace2fio.o 2026-02-26T03:12:51.502 INFO:teuthology.orchestra.run.trial053.stdout: LINK t/fio-btrace2fio 2026-02-26T03:12:51.555 INFO:teuthology.orchestra.run.trial053.stdout: CC t/verify-state.o 2026-02-26T03:12:51.616 INFO:teuthology.orchestra.run.trial053.stdout: CC t/debug.o 2026-02-26T03:12:51.642 INFO:teuthology.orchestra.run.trial053.stdout: LINK t/fio-verify-state 2026-02-26T03:12:51.695 INFO:teuthology.orchestra.run.trial053.stdout: CC t/stest.o 2026-02-26T03:12:51.746 INFO:teuthology.orchestra.run.trial053.stdout: CC t/arch.o 2026-02-26T03:12:51.769 INFO:teuthology.orchestra.run.trial053.stdout: LINK t/stest 2026-02-26T03:12:51.824 INFO:teuthology.orchestra.run.trial053.stdout: CC t/ieee754.o 2026-02-26T03:12:51.852 INFO:teuthology.orchestra.run.trial053.stdout: LINK t/ieee754 2026-02-26T03:12:51.905 INFO:teuthology.orchestra.run.trial053.stdout: CC t/axmap.o 2026-02-26T03:12:51.962 INFO:teuthology.orchestra.run.trial053.stdout: LINK t/axmap 2026-02-26T03:12:52.014 INFO:teuthology.orchestra.run.trial053.stdout: CC t/lfsr-test.o 2026-02-26T03:12:52.071 INFO:teuthology.orchestra.run.trial053.stdout: LINK t/lfsr-test 2026-02-26T03:12:52.125 INFO:teuthology.orchestra.run.trial053.stdout: CC t/gen-rand.o 2026-02-26T03:12:52.181 INFO:teuthology.orchestra.run.trial053.stdout: LINK t/gen-rand 2026-02-26T03:12:52.235 INFO:teuthology.orchestra.run.trial053.stdout: CC t/memlock.o 2026-02-26T03:12:52.276 INFO:teuthology.orchestra.run.trial053.stdout: LINK t/memlock 2026-02-26T03:12:52.328 INFO:teuthology.orchestra.run.trial053.stdout: CC t/read-to-pipe-async.o 2026-02-26T03:12:52.429 INFO:teuthology.orchestra.run.trial053.stdout: LINK t/read-to-pipe-async 2026-02-26T03:12:52.483 INFO:teuthology.orchestra.run.trial053.stdout: CC t/io_uring.o 2026-02-26T03:12:52.755 INFO:teuthology.orchestra.run.trial053.stdout: LINK t/io_uring 2026-02-26T03:12:52.809 DEBUG:teuthology.orchestra.run.trial053:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2026-02-26T03:12:52.835 DEBUG:teuthology.orchestra.run.trial053:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2026-02-26T03:12:52.948 INFO:teuthology.orchestra.run.trial053.stderr:03:12:52 - DEBUG - cbt - Settings.cluster: 2026-02-26T03:12:52.948 INFO:teuthology.orchestra.run.trial053.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2026-02-26T03:12:52.949 INFO:teuthology.orchestra.run.trial053.stderr: 'clients': ['ubuntu@trial053.front.sepia.ceph.com'], 2026-02-26T03:12:52.949 INFO:teuthology.orchestra.run.trial053.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2026-02-26T03:12:52.949 INFO:teuthology.orchestra.run.trial053.stderr: 'head': 'ubuntu@trial053.front.sepia.ceph.com', 2026-02-26T03:12:52.949 INFO:teuthology.orchestra.run.trial053.stderr: 'is_teuthology': True, 2026-02-26T03:12:52.949 INFO:teuthology.orchestra.run.trial053.stderr: 'iterations': 1, 2026-02-26T03:12:52.949 INFO:teuthology.orchestra.run.trial053.stderr: 'mons': ['ubuntu@trial053.front.sepia.ceph.com'], 2026-02-26T03:12:52.949 INFO:teuthology.orchestra.run.trial053.stderr: 'osds': ['ubuntu@trial053.front.sepia.ceph.com'], 2026-02-26T03:12:52.949 INFO:teuthology.orchestra.run.trial053.stderr: 'osds_per_node': 3, 2026-02-26T03:12:52.949 INFO:teuthology.orchestra.run.trial053.stderr: 'pid_dir': '/var/run/ceph', 2026-02-26T03:12:52.949 INFO:teuthology.orchestra.run.trial053.stderr: 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 2026-02-26T03:12:52.950 INFO:teuthology.orchestra.run.trial053.stderr: 'rebuild_every_test': False, 2026-02-26T03:12:52.950 INFO:teuthology.orchestra.run.trial053.stderr: 'rgws': {'ubuntu@trial053.front.sepia.ceph.com': None}, 2026-02-26T03:12:52.950 INFO:teuthology.orchestra.run.trial053.stderr: 'tmp_dir': '/tmp/cbt', 2026-02-26T03:12:52.950 INFO:teuthology.orchestra.run.trial053.stderr: 'use_existing': True, 2026-02-26T03:12:52.950 INFO:teuthology.orchestra.run.trial053.stderr: 'user': 'ubuntu'} 2026-02-26T03:12:52.950 INFO:teuthology.orchestra.run.trial053.stderr:03:12:52 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be 2026-02-26T03:12:52.950 INFO:teuthology.orchestra.run.trial053.stderr:03:12:52 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com if [ -d "/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw" ]; then rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw; fi 2026-02-26T03:12:53.131 INFO:teuthology.orchestra.run.trial053.stderr:03:12:53 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw 2026-02-26T03:12:53.295 INFO:teuthology.orchestra.run.trial053.stderr:03:12:53 - INFO - cbt - Pausing for 60s for idle monitoring. 2026-02-26T03:12:53.295 INFO:teuthology.orchestra.run.trial053.stderr:03:12:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/idle_monitoring/collectl 2026-02-26T03:12:53.461 INFO:teuthology.orchestra.run.trial053.stderr:03:12:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-26T03:12:53.463 INFO:teuthology.orchestra.run.trial053.stderr:03:12:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/idle_monitoring/perf 2026-02-26T03:12:53.634 INFO:teuthology.orchestra.run.trial053.stderr:03:12:53 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-26T03:12:53.634 INFO:teuthology.orchestra.run.trial053.stderr:03:12:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/idle_monitoring/perf/perf_stat.${pid} & done 2026-02-26T03:13:53.695 INFO:teuthology.orchestra.run.trial053.stderr:03:13:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-26T03:13:53.870 INFO:teuthology.orchestra.run.trial053.stderr:03:13:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-26T03:13:54.057 INFO:teuthology.orchestra.run.trial053.stderr:03:13:54 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/* 2026-02-26T03:13:54.234 INFO:teuthology.orchestra.run.trial053.stderr:03:13:54 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial053.front.sepia.ceph.com -r /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be 2026-02-26T03:13:54.401 INFO:teuthology.orchestra.run.trial053.stderr:03:13:54 - INFO - cbt - Creating fio images... 2026-02-26T03:13:54.402 INFO:teuthology.orchestra.run.trial053.stderr:03:13:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/pool_monitoring/collectl 2026-02-26T03:13:54.566 INFO:teuthology.orchestra.run.trial053.stderr:03:13:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-26T03:13:54.567 INFO:teuthology.orchestra.run.trial053.stderr:03:13:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/pool_monitoring/perf 2026-02-26T03:13:54.738 INFO:teuthology.orchestra.run.trial053.stderr:03:13:54 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-26T03:13:54.738 INFO:teuthology.orchestra.run.trial053.stderr:03:13:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/pool_monitoring/perf/perf_stat.${pid} & done 2026-02-26T03:13:54.740 INFO:teuthology.orchestra.run.trial053.stderr:03:13:54 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete cbt-librbdfio cbt-librbdfio --yes-i-really-really-mean-it 2026-02-26T03:13:55.088 INFO:teuthology.orchestra.run.trial053.stderr:03:13:55 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create cbt-librbdfio 128 128 2026-02-26T03:13:55.950 INFO:teuthology.orchestra.run.trial053.stderr:03:13:55 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable cbt-librbdfio rbd 2026-02-26T03:13:56.954 INFO:teuthology.orchestra.run.trial053.stderr:03:13:56 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio size 3 2026-02-26T03:13:57.963 INFO:teuthology.orchestra.run.trial053.stderr:03:13:57 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio min_size 2 2026-02-26T03:13:58.961 INFO:teuthology.orchestra.run.trial053.stderr:03:13:58 - INFO - cbt - Checking Health after pool creation. 2026-02-26T03:14:03.966 INFO:teuthology.orchestra.run.trial053.stderr:03:14:03 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com /usr/bin/ceph -c /etc/ceph/ceph.conf health 2026-02-26T03:14:04.326 INFO:teuthology.orchestra.run.trial053.stderr:03:14:04 - INFO - cbt - Final Pool Health Check. 2026-02-26T03:14:09.332 INFO:teuthology.orchestra.run.trial053.stderr:03:14:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com /usr/bin/ceph -c /etc/ceph/ceph.conf health 2026-02-26T03:14:09.697 INFO:teuthology.orchestra.run.trial053.stderr:03:14:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com hostname -f 2026-02-26T03:14:09.866 INFO:teuthology.orchestra.run.trial053.stderr:03:14:09 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com /usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-trial053-0 --size 4096 --pool cbt-librbdfio --order 22 2026-02-26T03:14:12.073 INFO:teuthology.orchestra.run.trial053.stderr:03:14:12 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com /usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-trial053-1 --size 4096 --pool cbt-librbdfio --order 22 2026-02-26T03:14:12.331 INFO:teuthology.orchestra.run.trial053.stderr:03:14:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-26T03:14:12.505 INFO:teuthology.orchestra.run.trial053.stderr:03:14:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-26T03:14:12.687 INFO:teuthology.orchestra.run.trial053.stderr:03:14:12 - INFO - cbt - Attempting to prefill fio images... 2026-02-26T03:14:12.687 INFO:teuthology.orchestra.run.trial053.stderr:03:14:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2026-02-26T03:14:12.689 INFO:teuthology.orchestra.run.trial053.stderr:03:14:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2026-02-26T03:14:52.284 INFO:teuthology.orchestra.run.trial053.stderr:03:14:52 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2026-02-26T03:14:52.285 INFO:teuthology.orchestra.run.trial053.stderr:03:14:52 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2026-02-26T03:15:15.354 INFO:teuthology.orchestra.run.trial053.stderr:03:15:15 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be 2026-02-26T03:15:15.354 INFO:teuthology.orchestra.run.trial053.stderr:03:15:15 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be. 2026-02-26T03:15:15.355 INFO:teuthology.orchestra.run.trial053.stderr:03:15:15 - INFO - cbt - Running benchmark /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw 2026-02-26T03:15:15.355 INFO:teuthology.orchestra.run.trial053.stderr:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be 2026-02-26T03:15:15.355 INFO:teuthology.orchestra.run.trial053.stderr:{'cmd_path': '/home/ubuntu/cephtest/fio/fio', 'log_avg_msec': 100, 'norandommap': True, 'pool_profile': 'rbd', 'time': 60, 'vol_size': 4096, 'iodepth': 32, 'mode': 'randrw', 'op_size': 4096, 'osd_ra': 4096, 'procs_per_volume': 1, 'volumes_per_client': 2, 'benchmark': 'librbdfio', 'iteration': 0} == iteration 0 == 2026-02-26T03:15:15.355 INFO:teuthology.orchestra.run.trial053.stderr:03:15:15 - INFO - cbt - Setting OSD Read Ahead to: 4096 2026-02-26T03:15:15.355 INFO:teuthology.orchestra.run.trial053.stderr:03:15:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com find /dev/disk/by-partlabel/osd-device-*data -exec readlink {} \; | cut -d"/" -f 3 | sed "s/[0-9]$//" | xargs -I{} sudo sh -c "echo 4096 > /sys/block/'{}'/queue/read_ahead_kb" 2026-02-26T03:15:15.356 INFO:teuthology.orchestra.run.trial053.stderr:03:15:15 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw 2026-02-26T03:15:15.356 INFO:teuthology.orchestra.run.trial053.stderr:03:15:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com sudo rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw 2026-02-26T03:15:15.532 INFO:teuthology.orchestra.run.trial053.stderr:03:15:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com sync 2026-02-26T03:15:15.713 INFO:teuthology.orchestra.run.trial053.stderr:03:15:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com echo 3 | sudo tee /proc/sys/vm/drop_caches 2026-02-26T03:15:16.893 INFO:teuthology.orchestra.run.trial053.stderr:03:15:16 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw 2026-02-26T03:15:17.088 INFO:teuthology.orchestra.run.trial053.stderr:03:15:17 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/ceph_settings.out 2026-02-26T03:15:42.407 INFO:teuthology.orchestra.run.trial053.stderr:03:15:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com echo "[`date`] ceph progress" >> /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/pgautoscaler.log;/usr/bin/ceph -c /etc/ceph/ceph.conf progress | tee -a /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/pgautoscaler.log 2026-02-26T03:15:42.774 INFO:teuthology.orchestra.run.trial053.stderr:03:15:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/collectl 2026-02-26T03:15:42.939 INFO:teuthology.orchestra.run.trial053.stderr:03:15:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-26T03:15:42.941 INFO:teuthology.orchestra.run.trial053.stderr:03:15:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/perf 2026-02-26T03:15:43.116 INFO:teuthology.orchestra.run.trial053.stderr:03:15:43 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-26T03:15:43.116 INFO:teuthology.orchestra.run.trial053.stderr:03:15:43 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/perf/perf_stat.${pid} & done 2026-02-26T03:15:43.117 INFO:teuthology.orchestra.run.trial053.stderr:03:15:43 - INFO - cbt - Running rbd fio randrw test. 2026-02-26T03:15:43.117 INFO:teuthology.orchestra.run.trial053.stderr:03:15:43 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-0 2026-02-26T03:15:43.118 INFO:teuthology.orchestra.run.trial053.stderr:03:15:43 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=randrw --output-format=json,normal --rwmixread=50 --rwmixwrite=50 --runtime=60 --numjobs=1 --direct=1 --bs=4096B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/output.0 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/output.0 2026-02-26T03:15:43.119 INFO:teuthology.orchestra.run.trial053.stderr:03:15:43 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-1 2026-02-26T03:15:43.119 INFO:teuthology.orchestra.run.trial053.stderr:03:15:43 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=randrw --output-format=json,normal --rwmixread=50 --rwmixwrite=50 --runtime=60 --numjobs=1 --direct=1 --bs=4096B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/output.1 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/output.1 2026-02-26T03:16:43.866 INFO:teuthology.orchestra.run.trial053.stderr:03:16:43 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-26T03:16:44.051 INFO:teuthology.orchestra.run.trial053.stderr:03:16:44 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-26T03:16:44.234 INFO:teuthology.orchestra.run.trial053.stderr:03:16:44 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/perf/perf.data 2026-02-26T03:16:44.236 INFO:teuthology.orchestra.run.trial053.stderr:03:16:44 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/perf/perf_stat.* 2026-02-26T03:16:44.238 INFO:teuthology.orchestra.run.trial053.stderr:03:16:44 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/historic_ops.out 2026-02-26T03:16:44.599 INFO:teuthology.orchestra.run.trial053.stderr:03:16:44 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial053.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/* 2026-02-26T03:16:44.771 INFO:teuthology.orchestra.run.trial053.stderr:03:16:44 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial053.front.sepia.ceph.com -r /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be 2026-02-26T03:16:44.939 INFO:teuthology.orchestra.run.trial053.stderr:03:16:44 - INFO - cbt - Convert results to json format. 2026-02-26T03:16:44.939 INFO:teuthology.orchestra.run.trial053.stdout:cleaning remote dir /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw 2026-02-26T03:16:44.940 INFO:teuthology.orchestra.run.trial053.stdout:trial053: trial053 2026-02-26T03:16:44.940 INFO:teuthology.orchestra.run.trial053.stdout: 2026-02-26T03:16:44.940 INFO:teuthology.orchestra.run.trial053.stdout:['trial053'] 2026-02-26T03:16:44.950 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2026-02-26T03:16:44.968 DEBUG:teuthology.orchestra.run.trial053:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2026-02-26T03:16:44.993 DEBUG:teuthology.orchestra.run.trial053:> rm --one-file-system -rf -- /home/ubuntu/cephtest/fio 2026-02-26T03:16:45.073 INFO:tasks.cbt_performance:Collecting CBT performance data config 2026-02-26T03:16:45.074 INFO:tasks.cbt_performance:reading cbt preferences 2026-02-26T03:16:45.074 DEBUG:teuthology.orchestra.run.trial053:> find /home/ubuntu/cephtest -name 'json_output.*' 2026-02-26T03:16:45.132 INFO:teuthology.orchestra.run.trial053.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be/json_output.1.trial053.front.sepia.ceph.com 2026-02-26T03:16:45.132 INFO:teuthology.orchestra.run.trial053.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be/json_output.0.trial053.front.sepia.ceph.com 2026-02-26T03:16:45.133 DEBUG:teuthology.orchestra.run.trial053:> set -ex 2026-02-26T03:16:45.133 DEBUG:teuthology.orchestra.run.trial053:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be/json_output.1.trial053.front.sepia.ceph.com of=/dev/stdout 2026-02-26T03:16:45.190 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-26T03:16:45.191 DEBUG:teuthology.orchestra.run.trial053:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be -name 'perf_stat.*' 2026-02-26T03:16:45.247 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-26T03:16:45.247 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-26T03:16:45.247 DEBUG:teuthology.orchestra.run.trial053:> set -ex 2026-02-26T03:16:45.247 DEBUG:teuthology.orchestra.run.trial053:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be/json_output.0.trial053.front.sepia.ceph.com of=/dev/stdout 2026-02-26T03:16:45.304 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-26T03:16:45.305 DEBUG:teuthology.orchestra.run.trial053:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be -name 'perf_stat.*' 2026-02-26T03:16:45.361 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-26T03:16:45.361 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-26T03:16:45.364 WARNING:tasks.cbt_performance:Could not send performance data to http://mira118.front.sepia.ceph.com:4000/cbt_performance: HTTPConnectionPool(host='mira118.front.sepia.ceph.com', port=4000): Max retries exceeded with url: /cbt_performance (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -2] Name or service not known')) 2026-02-26T03:16:45.364 WARNING:tasks.cbt_performance:Continuing without performance data collection 2026-02-26T03:16:45.364 DEBUG:teuthology.run_tasks:Unwinding manager ssh_keys 2026-02-26T03:16:45.375 INFO:teuthology.task.ssh_keys:Cleaning up SSH keys 2026-02-26T03:16:45.376 INFO:teuthology.task.ssh_keys:cleaning up keys added for testing 2026-02-26T03:16:45.376 INFO:teuthology.task.ssh_keys: cleaning up keys for user ubuntu on trial053.front.sepia.ceph.com 2026-02-26T03:16:45.376 DEBUG:teuthology.orchestra.run.trial053:> rm -- /home/ubuntu/.ssh/id_rsa 2026-02-26T03:16:45.417 DEBUG:teuthology.orchestra.run.trial053:> rm -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-26T03:16:45.472 DEBUG:teuthology.orchestra.run.trial053:> stat -c '"%a"' /home/ubuntu/.ssh/authorized_keys 2026-02-26T03:16:45.529 INFO:teuthology.orchestra.run.trial053.stdout:"600" 2026-02-26T03:16:45.529 DEBUG:teuthology.orchestra.run.trial053:> mv -- /home/ubuntu/.ssh/authorized_keys_2026-02-26_03:12:21:856986 /home/ubuntu/.ssh/authorized_keys 2026-02-26T03:16:45.587 DEBUG:teuthology.orchestra.run.trial053:> chmod 600 /home/ubuntu/.ssh/authorized_keys 2026-02-26T03:16:45.643 DEBUG:teuthology.orchestra.run.trial053:> rm /home/ubuntu/.ssh/config 2026-02-26T03:16:45.699 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-26T03:16:45.710 DEBUG:teuthology.orchestra.run.trial053:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-26T03:16:45.815 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:16:45.809+0000 7f419be5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:16:45.816 INFO:teuthology.orchestra.run.trial053.stderr:2026-02-26T03:16:45.814+0000 7f419be5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:16:45.938 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-26T03:16:45.939 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-26T03:16:45.939 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-26T03:16:45.939 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:16:49.708 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2026-02-26T03:16:52.041 INFO:tasks.ceph.osd.0:Stopped 2026-02-26T03:16:52.041 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-26T03:16:52.042 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:16:55.010 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2026-02-26T03:16:55.010 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2026-02-26T03:16:58.143 INFO:tasks.ceph.osd.1:Stopped 2026-02-26T03:16:58.143 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-26T03:16:58.144 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:17:00.212 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2026-02-26T03:17:00.212 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2026-02-26T03:17:04.245 INFO:tasks.ceph.osd.2:Stopped 2026-02-26T03:17:04.246 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-26T03:17:04.246 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-26T03:17:04.246 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:17:04.284 INFO:tasks.ceph.mgr.x:Stopped 2026-02-26T03:17:04.284 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-26T03:17:04.284 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-26T03:17:04.284 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:17:04.333 INFO:tasks.ceph.mon.a:Stopped 2026-02-26T03:17:04.333 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-26T03:17:04.333 DEBUG:teuthology.orchestra.run.trial053:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | grep -E -v '\(PG_' | grep -E -v '\(OSD_' | grep -E -v '\(OBJECT_' | grep -E -v 'overall HEALTH' | head -n 1 2026-02-26T03:17:04.399 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial053.front.sepia.ceph.com 2026-02-26T03:17:04.399 DEBUG:teuthology.orchestra.run.trial053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-26T03:17:04.475 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial053.front.sepia.ceph.com 2026-02-26T03:17:04.475 DEBUG:teuthology.orchestra.run.trial053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-26T03:17:04.546 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial053.front.sepia.ceph.com 2026-02-26T03:17:04.546 DEBUG:teuthology.orchestra.run.trial053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-26T03:17:04.616 INFO:tasks.ceph:Archiving mon data... 2026-02-26T03:17:04.618 DEBUG:teuthology.misc:Transferring archived files from trial053:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71147/data/mon.a.tgz 2026-02-26T03:17:04.618 DEBUG:teuthology.orchestra.run.trial053:> mktemp 2026-02-26T03:17:04.672 INFO:teuthology.orchestra.run.trial053.stdout:/tmp/tmp.qT59v6PfOe 2026-02-26T03:17:04.672 DEBUG:teuthology.orchestra.run.trial053:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.qT59v6PfOe 2026-02-26T03:17:04.762 DEBUG:teuthology.orchestra.run.trial053:> sudo chmod 0666 /tmp/tmp.qT59v6PfOe 2026-02-26T03:17:04.844 DEBUG:teuthology.orchestra.remote:trial053:/tmp/tmp.qT59v6PfOe is 208KB 2026-02-26T03:17:04.938 DEBUG:teuthology.orchestra.run.trial053:> rm -fr /tmp/tmp.qT59v6PfOe 2026-02-26T03:17:04.954 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-26T03:17:04.954 DEBUG:teuthology.orchestra.run.trial053:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-26T03:17:05.076 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-26T03:17:05.076 INFO:tasks.ceph:Archiving crash dumps... 2026-02-26T03:17:05.081 DEBUG:teuthology.misc:Transferring archived files from trial053:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71147/remote/trial053/crash 2026-02-26T03:17:05.083 DEBUG:teuthology.orchestra.run.trial053:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-26T03:17:05.139 INFO:tasks.ceph:Compressing logs... 2026-02-26T03:17:05.140 DEBUG:teuthology.orchestra.run.trial053:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-26T03:17:05.203 INFO:teuthology.orchestra.run.trial053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-26T03:17:05.216 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2026-02-26T03:17:05.218 INFO:teuthology.orchestra.run.trial053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41611.log 2026-02-26T03:17:05.218 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-26T03:17:05.218 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.41611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40388.log 2026-02-26T03:17:05.218 INFO:teuthology.orchestra.run.trial053.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41611.log.gz 2026-02-26T03:17:05.218 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-26T03:17:05.218 INFO:teuthology.orchestra.run.trial053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40525.log 2026-02-26T03:17:05.219 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.40388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48328.log 2026-02-26T03:17:05.219 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-osd.0.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40388.log.gz 2026-02-26T03:17:05.219 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.40525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40838.log 2026-02-26T03:17:05.219 INFO:teuthology.orchestra.run.trial053.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-26T03:17:05.219 INFO:teuthology.orchestra.run.trial053.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40525.log.gz 2026-02-26T03:17:05.219 INFO:teuthology.orchestra.run.trial053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47272.log 2026-02-26T03:17:05.219 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.48328.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48328.log.gz 2026-02-26T03:17:05.219 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.40838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47358.log 2026-02-26T03:17:05.219 INFO:teuthology.orchestra.run.trial053.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40838.log.gz 2026-02-26T03:17:05.219 INFO:teuthology.orchestra.run.trial053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47406.log 2026-02-26T03:17:05.219 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.47272.log: /var/log/ceph/ceph-client.admin.47358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47584.log 2026-02-26T03:17:05.219 INFO:teuthology.orchestra.run.trial053.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47272.log.gz 2026-02-26T03:17:05.220 INFO:teuthology.orchestra.run.trial053.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.47358.log.gz 2026-02-26T03:17:05.220 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.47406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41830.log 2026-02-26T03:17:05.220 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.47584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40572.log 2026-02-26T03:17:05.220 INFO:teuthology.orchestra.run.trial053.stderr: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.47406.log.gz 2026-02-26T03:17:05.220 INFO:teuthology.orchestra.run.trial053.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47584.log.gz 2026-02-26T03:17:05.220 INFO:teuthology.orchestra.run.trial053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41732.log 2026-02-26T03:17:05.220 INFO:teuthology.orchestra.run.trial053.stderr:gzip/var/log/ceph/ceph-client.admin.40572.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40869.log 2026-02-26T03:17:05.220 INFO:teuthology.orchestra.run.trial053.stderr: 47.6% -- replaced with /var/log/ceph/ceph-client.admin.40572.log.gz 2026-02-26T03:17:05.220 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.41732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47758.log 2026-02-26T03:17:05.220 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.41830.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41732.log.gz 2026-02-26T03:17:05.220 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.40869.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41830.log.gz 2026-02-26T03:17:05.220 INFO:teuthology.orchestra.run.trial053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48327.log 2026-02-26T03:17:05.221 INFO:teuthology.orchestra.run.trial053.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40869.log.gz 2026-02-26T03:17:05.221 INFO:teuthology.orchestra.run.trial053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47183.log 2026-02-26T03:17:05.221 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.47758.log: /var/log/ceph/ceph-client.admin.48327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41015.log 2026-02-26T03:17:05.221 INFO:teuthology.orchestra.run.trial053.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48327.log.gz 2026-02-26T03:17:05.221 INFO:teuthology.orchestra.run.trial053.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47758.log.gz 2026-02-26T03:17:05.221 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.47183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47069.log 2026-02-26T03:17:05.221 INFO:teuthology.orchestra.run.trial053.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47183.log.gz 2026-02-26T03:17:05.221 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.41015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41291.log 2026-02-26T03:17:05.221 INFO:teuthology.orchestra.run.trial053.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41015.log.gz 2026-02-26T03:17:05.221 INFO:teuthology.orchestra.run.trial053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41340.log 2026-02-26T03:17:05.221 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.47069.log: /var/log/ceph/ceph-client.admin.41291.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47069.log.gz 2026-02-26T03:17:05.221 INFO:teuthology.orchestra.run.trial053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47126.log 2026-02-26T03:17:05.221 INFO:teuthology.orchestra.run.trial053.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41291.log.gz 2026-02-26T03:17:05.222 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.41340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41639.log 2026-02-26T03:17:05.222 INFO:teuthology.orchestra.run.trial053.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41340.log.gz 2026-02-26T03:17:05.222 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.47126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40245.log 2026-02-26T03:17:05.222 INFO:teuthology.orchestra.run.trial053.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47126.log.gz 2026-02-26T03:17:05.222 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.41639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41492.log 2026-02-26T03:17:05.222 INFO:teuthology.orchestra.run.trial053.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41639.log.gz 2026-02-26T03:17:05.222 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.40245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47010.log 2026-02-26T03:17:05.222 INFO:teuthology.orchestra.run.trial053.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40245.log.gz 2026-02-26T03:17:05.222 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.41492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41781.log 2026-02-26T03:17:05.222 INFO:teuthology.orchestra.run.trial053.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41492.log.gz 2026-02-26T03:17:05.222 INFO:teuthology.orchestra.run.trial053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.38956.log 2026-02-26T03:17:05.222 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.47010.log: /var/log/ceph/ceph-client.admin.41781.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47010.log.gz 2026-02-26T03:17:05.222 INFO:teuthology.orchestra.run.trial053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48092.log 2026-02-26T03:17:05.223 INFO:teuthology.orchestra.run.trial053.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41781.log.gz 2026-02-26T03:17:05.223 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph.tmp-client.admin.38956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41082.log 2026-02-26T03:17:05.223 INFO:teuthology.orchestra.run.trial053.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.38956.log.gz 2026-02-26T03:17:05.223 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.48092.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-26T03:17:05.223 INFO:teuthology.orchestra.run.trial053.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48092.log.gz 2026-02-26T03:17:05.223 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.41082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41177.log 2026-02-26T03:17:05.223 INFO:teuthology.orchestra.run.trial053.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41082.log.gz 2026-02-26T03:17:05.223 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47218.log 2026-02-26T03:17:05.223 INFO:teuthology.orchestra.run.trial053.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.41177.log: -- /var/log/ceph/ceph-client.admin.41491.log 2026-02-26T03:17:05.223 INFO:teuthology.orchestra.run.trial053.stderr: 86.4%/var/log/ceph/ceph-client.admin.47218.log: -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-26T03:17:05.223 INFO:teuthology.orchestra.run.trial053.stderr: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.41177.log.gz 2026-02-26T03:17:05.224 INFO:teuthology.orchestra.run.trial053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-26T03:17:05.224 INFO:teuthology.orchestra.run.trial053.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47218.log.gz 2026-02-26T03:17:05.224 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.41491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41389.log 2026-02-26T03:17:05.224 INFO:teuthology.orchestra.run.trial053.stderr: 48.4% -- replaced with /var/log/ceph/ceph-client.admin.41491.log.gz 2026-02-26T03:17:05.224 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48993.log 2026-02-26T03:17:05.224 INFO:teuthology.orchestra.run.trial053.stderr:gzip/var/log/ceph/ceph-client.admin.41389.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41152.log 2026-02-26T03:17:05.224 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.48993.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41389.log.gz 2026-02-26T03:17:05.224 INFO:teuthology.orchestra.run.trial053.stderr:gzip 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48993.log.gz -5 2026-02-26T03:17:05.224 INFO:teuthology.orchestra.run.trial053.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47582.log 2026-02-26T03:17:05.224 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.41152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41242.log 2026-02-26T03:17:05.225 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.47582.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-26T03:17:05.225 INFO:teuthology.orchestra.run.trial053.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41152.log.gz 2026-02-26T03:17:05.225 INFO:teuthology.orchestra.run.trial053.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47582.log.gz 2026-02-26T03:17:05.225 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.41242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41612.log 2026-02-26T03:17:05.225 INFO:teuthology.orchestra.run.trial053.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41242.log.gz 2026-02-26T03:17:05.225 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46954.log 2026-02-26T03:17:05.225 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.41612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41490.log 2026-02-26T03:17:05.225 INFO:teuthology.orchestra.run.trial053.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41612.log.gz 2026-02-26T03:17:05.225 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.46954.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46954.log.gz 2026-02-26T03:17:05.225 INFO:teuthology.orchestra.run.trial053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40619.log 2026-02-26T03:17:05.225 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.41490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47759.log 2026-02-26T03:17:05.225 INFO:teuthology.orchestra.run.trial053.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41490.log.gz 2026-02-26T03:17:05.225 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.40619.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-26T03:17:05.226 INFO:teuthology.orchestra.run.trial053.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40619.log.gz 2026-02-26T03:17:05.226 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph-client.admin.47759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40341.log 2026-02-26T03:17:05.226 INFO:teuthology.orchestra.run.trial053.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47759.log.gz 2026-02-26T03:17:05.226 INFO:teuthology.orchestra.run.trial053.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.40341.log: 87.0% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-26T03:17:05.226 INFO:teuthology.orchestra.run.trial053.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40341.log.gz 2026-02-26T03:17:05.226 INFO:teuthology.orchestra.run.trial053.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-26T03:17:12.992 INFO:teuthology.orchestra.run.trial053.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-26T03:17:13.080 INFO:teuthology.orchestra.run.trial053.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-26T03:17:13.755 INFO:teuthology.orchestra.run.trial053.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-26T03:17:13.757 INFO:teuthology.orchestra.run.trial053.stderr: 2026-02-26T03:17:13.757 INFO:teuthology.orchestra.run.trial053.stderr:real 0m8.561s 2026-02-26T03:17:13.757 INFO:teuthology.orchestra.run.trial053.stderr:user 0m22.944s 2026-02-26T03:17:13.757 INFO:teuthology.orchestra.run.trial053.stderr:sys 0m1.257s 2026-02-26T03:17:13.757 INFO:tasks.ceph:Archiving logs... 2026-02-26T03:17:13.758 DEBUG:teuthology.misc:Transferring archived files from trial053:/var/log/ceph to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71147/remote/trial053/log 2026-02-26T03:17:13.759 DEBUG:teuthology.orchestra.run.trial053:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-26T03:17:15.889 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-26T03:17:15.984 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-26T03:17:15.996 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-26T03:17:15.996 DEBUG:teuthology.orchestra.run.trial053:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-26T03:17:16.021 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-26T03:17:16.033 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-26T03:17:16.033 DEBUG:teuthology.orchestra.run.trial053:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-26T03:17:16.077 INFO:teuthology.orchestra.run.trial053.stderr:bash: line 1: ntpq: command not found 2026-02-26T03:17:16.098 INFO:teuthology.orchestra.run.trial053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-26T03:17:16.098 INFO:teuthology.orchestra.run.trial053.stdout:=============================================================================== 2026-02-26T03:17:16.098 INFO:teuthology.orchestra.run.trial053.stdout:^- ntp.shastacoe.net 2 6 377 8 -443us[ -443us] +/- 87ms 2026-02-26T03:17:16.098 INFO:teuthology.orchestra.run.trial053.stdout:^* vps-lga1.orleans.ddnss.de 2 6 377 9 +22us[ +35us] +/- 2218us 2026-02-26T03:17:16.098 INFO:teuthology.orchestra.run.trial053.stdout:^- 23.186.168.130 2 6 335 7 +814us[ +814us] +/- 56ms 2026-02-26T03:17:16.098 INFO:teuthology.orchestra.run.trial053.stdout:^- tick.srs1.ntfo.org 3 6 7 4 +994us[ +994us] +/- 122ms 2026-02-26T03:17:16.099 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-26T03:17:16.114 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-26T03:17:16.114 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-26T03:17:16.128 DEBUG:teuthology.orchestra.run.trial053:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-26T03:17:16.198 DEBUG:teuthology.orchestra.run.trial053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-26T03:17:16.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:17:16.227 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-26T03:17:16.244 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-26T03:17:16.255 INFO:teuthology.task.internal:Duration was 821.374668 seconds 2026-02-26T03:17:16.255 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-26T03:17:16.269 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-26T03:17:16.269 DEBUG:teuthology.orchestra.run.trial053:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-26T03:17:16.307 INFO:teuthology.orchestra.run.trial053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-26T03:17:16.615 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-26T03:17:16.616 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial053.front.sepia.ceph.com 2026-02-26T03:17:16.616 DEBUG:teuthology.orchestra.run.trial053:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-26T03:17:16.675 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-26T03:17:16.675 DEBUG:teuthology.orchestra.run.trial053:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-26T03:17:16.778 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-26T03:17:16.779 DEBUG:teuthology.orchestra.run.trial053:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-26T03:17:16.801 INFO:teuthology.orchestra.run.trial053.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-26T03:17:16.801 INFO:teuthology.orchestra.run.trial053.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-26T03:17:16.802 INFO:teuthology.orchestra.run.trial053.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-26T03:17:16.802 INFO:teuthology.orchestra.run.trial053.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-26T03:17:16.802 INFO:teuthology.orchestra.run.trial053.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-26T03:17:16.812 INFO:teuthology.orchestra.run.trial053.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 82.6% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-26T03:17:16.815 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-26T03:17:16.834 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-26T03:17:16.834 DEBUG:teuthology.orchestra.run.trial053:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-26T03:17:16.879 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-26T03:17:16.898 DEBUG:teuthology.orchestra.run.trial053:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-26T03:17:16.945 INFO:teuthology.orchestra.run.trial053.stdout:kernel.core_pattern = core 2026-02-26T03:17:16.960 DEBUG:teuthology.orchestra.run.trial053:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-26T03:17:17.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:17:17.017 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-26T03:17:17.037 INFO:teuthology.task.internal:Transferring archived files... 2026-02-26T03:17:17.038 DEBUG:teuthology.misc:Transferring archived files from trial053:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71147/remote/trial053 2026-02-26T03:17:17.038 DEBUG:teuthology.orchestra.run.trial053:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-26T03:17:17.123 INFO:teuthology.task.internal:Removing archive directory... 2026-02-26T03:17:17.123 DEBUG:teuthology.orchestra.run.trial053:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-26T03:17:17.151 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-26T03:17:17.167 INFO:teuthology.task.internal:Not uploading archives. 2026-02-26T03:17:17.167 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-26T03:17:17.181 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-26T03:17:17.182 DEBUG:teuthology.orchestra.run.trial053:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-26T03:17:17.208 INFO:teuthology.orchestra.run.trial053.stdout: 920834 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 26 03:17 /home/ubuntu/cephtest 2026-02-26T03:17:17.210 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-26T03:17:17.230 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-26T03:17:17.271 INFO:teuthology.run:Summary data: description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm} perf_ignorelist settings/optimized workloads/fio_4K_rand_rw} duration: 821.3746676445007 flavor: debug owner: scheduled_shraddhaag@soko04.front.sepia.ceph.com success: true 2026-02-26T03:17:17.271 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-26T03:17:17.412 INFO:teuthology.run:pass