2026-02-26T02:36:04.917 INFO:root:teuthology version: 1.2.4.dev11+g6e3d0419c 2026-02-26T02:36:04.930 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-26T02:36:05.146 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/yuriw-2026-02-26_01:51:16-rados-wip-yuri9-testing-2026-02-25-1600-squid-distro-default-trial/71042 branch: wip-yuri9-testing-2026-02-25-1600-squid description: rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} email: yweinste@redhat.com first_in_suite: false flavor: default job_id: '71042' kernel: branch: distro kdb: true sha1: distro last_in_suite: false machine_type: trial name: yuriw-2026-02-26_01:51:16-rados-wip-yuri9-testing-2026-02-25-1600-squid-distro-default-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri9-testing-2026-02-25-1600-squid ceph: conf: global: mon client directed command retry: 5 mon client hunt interval max multiple: 2 mon election default strategy: 3 mon mgr beacon grace: 90 ms bind msgr2: false ms inject socket failures: 1000 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - \(MON_DOWN\) mon_bind_msgr2: false sha1: 8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri9-testing-2026-02-25-1600-squid sha1: 8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5 owner: scheduled_yuriw@soko04.front.sepia.ceph.com priority: 69 repo: https://github.com/ceph/ceph-ci.git roles: - - client.0 - mon.a - mgr.x - osd.0 - osd.1 - osd.2 seed: 1525 sha1: 8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri9-testing-2026-02-25-1600-squid suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5 targets: trial043.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPri+9SLxA35ppeGNRG0esqvoxCp406uXTc4O1OSKl3xWTZxdp+Ayg0GMvcRXGSupYz8guymaqfPqIjHwCoN/Gc= tasks: - install: null - ceph: log-ignorelist: - Reduced data availability - OSD_.*DOWN - \(RECENT_CRASH\) - \(POOL_APP_NOT_ENABLED\) - \(PG_DEGRADED\) pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force - workunit: clients: client.0: - rados/test_crash.sh - ceph.restart: - osd.* - exec: mon.a: - find $TESTDIR/archive/coredump -type f -exec rm -f {} \; teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6e3d0419ccca0e75a03e08feb708946e198236f4 timestamp: 2026-02-26_01:51:16 tube: trial user: yuriw verbose: true worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699 2026-02-26T02:36:05.146 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5/qa; will attempt to use it 2026-02-26T02:36:05.146 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5/qa/tasks 2026-02-26T02:36:05.147 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-26T02:36:05.148 INFO:teuthology.task.internal:Checking packages... 2026-02-26T02:36:05.148 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5' 2026-02-26T02:36:05.148 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-26T02:36:05.148 INFO:teuthology.packaging:ref: None 2026-02-26T02:36:05.148 INFO:teuthology.packaging:tag: None 2026-02-26T02:36:05.148 INFO:teuthology.packaging:branch: wip-yuri9-testing-2026-02-25-1600-squid 2026-02-26T02:36:05.148 INFO:teuthology.packaging:sha1: 8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5 2026-02-26T02:36:05.149 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri9-testing-2026-02-25-1600-squid 2026-02-26T02:36:05.264 INFO:teuthology.task.internal:Found packages for ceph version 19.2.3-687.g8d724284 2026-02-26T02:36:05.266 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-26T02:36:05.283 INFO:teuthology.task.internal:no buildpackages task found 2026-02-26T02:36:05.283 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-26T02:36:05.299 INFO:teuthology.task.internal:Saving configuration 2026-02-26T02:36:05.308 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-26T02:36:05.322 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-26T02:36:05.405 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial043.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-26_01:51:16-rados-wip-yuri9-testing-2026-02-25-1600-squid-distro-default-trial/71042', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-26 02:32:35.648511', 'locked_by': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPri+9SLxA35ppeGNRG0esqvoxCp406uXTc4O1OSKl3xWTZxdp+Ayg0GMvcRXGSupYz8guymaqfPqIjHwCoN/Gc='} 2026-02-26T02:36:05.405 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-26T02:36:05.424 INFO:teuthology.task.internal:roles: ubuntu@trial043.front.sepia.ceph.com - ['client.0', 'mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2'] 2026-02-26T02:36:05.424 INFO:teuthology.run_tasks:Running task console_log... 2026-02-26T02:36:05.623 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x711334821800>, signals=[15]) 2026-02-26T02:36:05.624 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-26T02:36:05.635 INFO:teuthology.task.internal:Opening connections... 2026-02-26T02:36:05.635 DEBUG:teuthology.task.internal:connecting to ubuntu@trial043.front.sepia.ceph.com 2026-02-26T02:36:05.637 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T02:36:05.713 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-26T02:36:05.724 DEBUG:teuthology.orchestra.run.trial043:> uname -m 2026-02-26T02:36:05.787 INFO:teuthology.orchestra.run.trial043.stdout:x86_64 2026-02-26T02:36:05.787 DEBUG:teuthology.orchestra.run.trial043:> cat /etc/os-release 2026-02-26T02:36:05.843 INFO:teuthology.orchestra.run.trial043.stdout:NAME="CentOS Stream" 2026-02-26T02:36:05.843 INFO:teuthology.orchestra.run.trial043.stdout:VERSION="9" 2026-02-26T02:36:05.843 INFO:teuthology.orchestra.run.trial043.stdout:ID="centos" 2026-02-26T02:36:05.843 INFO:teuthology.orchestra.run.trial043.stdout:ID_LIKE="rhel fedora" 2026-02-26T02:36:05.843 INFO:teuthology.orchestra.run.trial043.stdout:VERSION_ID="9" 2026-02-26T02:36:05.843 INFO:teuthology.orchestra.run.trial043.stdout:PLATFORM_ID="platform:el9" 2026-02-26T02:36:05.843 INFO:teuthology.orchestra.run.trial043.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-26T02:36:05.843 INFO:teuthology.orchestra.run.trial043.stdout:ANSI_COLOR="0;31" 2026-02-26T02:36:05.843 INFO:teuthology.orchestra.run.trial043.stdout:LOGO="fedora-logo-icon" 2026-02-26T02:36:05.843 INFO:teuthology.orchestra.run.trial043.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-26T02:36:05.843 INFO:teuthology.orchestra.run.trial043.stdout:HOME_URL="https://centos.org/" 2026-02-26T02:36:05.843 INFO:teuthology.orchestra.run.trial043.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-26T02:36:05.843 INFO:teuthology.orchestra.run.trial043.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-26T02:36:05.843 INFO:teuthology.orchestra.run.trial043.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-26T02:36:05.844 INFO:teuthology.lock.ops:Updating trial043.front.sepia.ceph.com on lock server 2026-02-26T02:36:05.928 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-26T02:36:05.940 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-26T02:36:05.952 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-26T02:36:05.952 DEBUG:teuthology.orchestra.run.trial043:> test '!' -e /home/ubuntu/cephtest 2026-02-26T02:36:05.967 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-26T02:36:05.977 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-26T02:36:05.977 DEBUG:teuthology.orchestra.run.trial043:> test -z $(ls -A /var/lib/ceph) 2026-02-26T02:36:06.023 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-26T02:36:06.095 INFO:teuthology.run_tasks:Running task kernel... 2026-02-26T02:36:06.115 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-26T02:36:06.115 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-26T02:36:06.115 DEBUG:teuthology.orchestra.run.trial043:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-26T02:36:06.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T02:36:06.130 DEBUG:teuthology.orchestra.run.trial043:> uname -r 2026-02-26T02:36:06.185 INFO:teuthology.orchestra.run.trial043.stdout:5.14.0-681.el9.x86_64 2026-02-26T02:36:06.185 INFO:teuthology.task.kernel:Running kernel on trial043: 5.14.0-681.el9.x86_64 2026-02-26T02:36:06.185 DEBUG:teuthology.orchestra.run.trial043:> sudo yum install -y kernel 2026-02-26T02:36:07.825 INFO:teuthology.orchestra.run.trial043.stdout:CentOS Stream 9 - BaseOS 7.1 MB/s | 8.9 MB 00:01 2026-02-26T02:36:10.889 INFO:teuthology.orchestra.run.trial043.stdout:CentOS Stream 9 - AppStream 11 MB/s | 27 MB 00:02 2026-02-26T02:36:14.456 INFO:teuthology.orchestra.run.trial043.stdout:CentOS Stream 9 - CRB 6.8 MB/s | 8.0 MB 00:01 2026-02-26T02:36:15.596 INFO:teuthology.orchestra.run.trial043.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 20 kB 00:00 2026-02-26T02:36:16.450 INFO:teuthology.orchestra.run.trial043.stdout:Extra Packages for Enterprise Linux 26 MB/s | 20 MB 00:00 2026-02-26T02:36:19.662 INFO:teuthology.orchestra.run.trial043.stdout:lab-extras 2.6 MB/s | 50 kB 00:00 2026-02-26T02:36:20.691 INFO:teuthology.orchestra.run.trial043.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-26T02:36:20.691 INFO:teuthology.orchestra.run.trial043.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-26T02:36:20.711 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-26T02:36:20.712 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-26T02:36:20.712 INFO:teuthology.orchestra.run.trial043.stdout: Package Architecture Version Repository Size 2026-02-26T02:36:20.712 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-26T02:36:20.712 INFO:teuthology.orchestra.run.trial043.stdout:Installing: 2026-02-26T02:36:20.713 INFO:teuthology.orchestra.run.trial043.stdout: kernel x86_64 5.14.0-686.el9 baseos 901 k 2026-02-26T02:36:20.713 INFO:teuthology.orchestra.run.trial043.stdout:Installing dependencies: 2026-02-26T02:36:20.713 INFO:teuthology.orchestra.run.trial043.stdout: kernel-core x86_64 5.14.0-686.el9 baseos 17 M 2026-02-26T02:36:20.713 INFO:teuthology.orchestra.run.trial043.stdout: kernel-modules x86_64 5.14.0-686.el9 baseos 40 M 2026-02-26T02:36:20.713 INFO:teuthology.orchestra.run.trial043.stdout: kernel-modules-core x86_64 5.14.0-686.el9 baseos 31 M 2026-02-26T02:36:20.713 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:36:20.713 INFO:teuthology.orchestra.run.trial043.stdout:Transaction Summary 2026-02-26T02:36:20.713 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-26T02:36:20.713 INFO:teuthology.orchestra.run.trial043.stdout:Install 4 Packages 2026-02-26T02:36:20.713 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:36:20.713 INFO:teuthology.orchestra.run.trial043.stdout:Total download size: 89 M 2026-02-26T02:36:20.714 INFO:teuthology.orchestra.run.trial043.stdout:Installed size: 135 M 2026-02-26T02:36:20.714 INFO:teuthology.orchestra.run.trial043.stdout:Downloading Packages: 2026-02-26T02:36:21.087 INFO:teuthology.orchestra.run.trial043.stdout:(1/4): kernel-5.14.0-686.el9.x86_64.rpm 6.9 MB/s | 901 kB 00:00 2026-02-26T02:36:21.598 INFO:teuthology.orchestra.run.trial043.stdout:(2/4): kernel-core-5.14.0-686.el9.x86_64.rpm 27 MB/s | 17 MB 00:00 2026-02-26T02:36:22.067 INFO:teuthology.orchestra.run.trial043.stdout:(3/4): kernel-modules-core-5.14.0-686.el9.x86_6 32 MB/s | 31 MB 00:00 2026-02-26T02:36:22.284 INFO:teuthology.orchestra.run.trial043.stdout:(4/4): kernel-modules-5.14.0-686.el9.x86_64.rpm 30 MB/s | 40 MB 00:01 2026-02-26T02:36:22.284 INFO:teuthology.orchestra.run.trial043.stdout:-------------------------------------------------------------------------------- 2026-02-26T02:36:22.284 INFO:teuthology.orchestra.run.trial043.stdout:Total 57 MB/s | 89 MB 00:01 2026-02-26T02:36:22.476 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction check 2026-02-26T02:36:22.678 INFO:teuthology.orchestra.run.trial043.stdout:Transaction check succeeded. 2026-02-26T02:36:22.679 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction test 2026-02-26T02:36:23.062 INFO:teuthology.orchestra.run.trial043.stdout:Transaction test succeeded. 2026-02-26T02:36:23.062 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction 2026-02-26T02:36:24.200 INFO:teuthology.orchestra.run.trial043.stdout: Preparing : 1/1 2026-02-26T02:36:24.504 INFO:teuthology.orchestra.run.trial043.stdout: Installing : kernel-modules-core-5.14.0-686.el9.x86_64 1/4 2026-02-26T02:36:24.527 INFO:teuthology.orchestra.run.trial043.stdout: Installing : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-26T02:36:27.094 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-26T02:36:27.128 INFO:teuthology.orchestra.run.trial043.stdout: Installing : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-26T02:36:30.229 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-26T02:36:30.235 INFO:teuthology.orchestra.run.trial043.stdout: Installing : kernel-5.14.0-686.el9.x86_64 4/4 2026-02-26T02:36:33.284 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-26T02:36:50.699 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 4/4 2026-02-26T02:36:50.720 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 4/4 2026-02-26T02:36:50.908 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: kernel-5.14.0-686.el9.x86_64 4/4 2026-02-26T02:36:50.908 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : kernel-5.14.0-686.el9.x86_64 1/4 2026-02-26T02:36:50.908 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-26T02:36:50.908 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-26T02:36:50.965 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-26T02:36:50.965 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:36:50.965 INFO:teuthology.orchestra.run.trial043.stdout:Installed: 2026-02-26T02:36:50.965 INFO:teuthology.orchestra.run.trial043.stdout: kernel-5.14.0-686.el9.x86_64 2026-02-26T02:36:50.965 INFO:teuthology.orchestra.run.trial043.stdout: kernel-core-5.14.0-686.el9.x86_64 2026-02-26T02:36:50.965 INFO:teuthology.orchestra.run.trial043.stdout: kernel-modules-5.14.0-686.el9.x86_64 2026-02-26T02:36:50.966 INFO:teuthology.orchestra.run.trial043.stdout: kernel-modules-core-5.14.0-686.el9.x86_64 2026-02-26T02:36:50.966 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:36:50.966 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-26T02:36:51.029 DEBUG:teuthology.orchestra.run.trial043:> rpm -q kernel | sort -rV | head -n 1 2026-02-26T02:36:51.058 INFO:teuthology.orchestra.run.trial043.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-26T02:36:51.059 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-26T02:36:51.059 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-681.el9.x86_64 Expected: 5.14.0-686.el9.x86_64 2026-02-26T02:36:51.059 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-26T02:36:51.059 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-26T02:36:51.059 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2026-02-26T02:36:51.059 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial043.front.sepia.ceph.com, path=None, version=distro) 2026-02-26T02:36:51.059 DEBUG:teuthology.orchestra.run.trial043:> rpm -q kernel | sort -rV | head -n 1 2026-02-26T02:36:51.125 INFO:teuthology.orchestra.run.trial043.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-26T02:36:51.126 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-26T02:36:51.126 DEBUG:teuthology.orchestra.run.trial043:> sudo rpm -qi grub2-tools 2026-02-26T02:36:51.194 INFO:teuthology.orchestra.run.trial043.stdout:Name : grub2-tools 2026-02-26T02:36:51.195 INFO:teuthology.orchestra.run.trial043.stdout:Epoch : 1 2026-02-26T02:36:51.195 INFO:teuthology.orchestra.run.trial043.stdout:Version : 2.06 2026-02-26T02:36:51.195 INFO:teuthology.orchestra.run.trial043.stdout:Release : 120.el9 2026-02-26T02:36:51.195 INFO:teuthology.orchestra.run.trial043.stdout:Architecture: x86_64 2026-02-26T02:36:51.195 INFO:teuthology.orchestra.run.trial043.stdout:Install Date: Wed 18 Feb 2026 05:18:42 PM UTC 2026-02-26T02:36:51.195 INFO:teuthology.orchestra.run.trial043.stdout:Group : Unspecified 2026-02-26T02:36:51.195 INFO:teuthology.orchestra.run.trial043.stdout:Size : 9108611 2026-02-26T02:36:51.195 INFO:teuthology.orchestra.run.trial043.stdout:License : GPLv3+ 2026-02-26T02:36:51.195 INFO:teuthology.orchestra.run.trial043.stdout:Signature : RSA/SHA256, Thu 04 Dec 2025 09:42:59 AM UTC, Key ID 05b555b38483c65d 2026-02-26T02:36:51.196 INFO:teuthology.orchestra.run.trial043.stdout:Source RPM : grub2-2.06-120.el9.src.rpm 2026-02-26T02:36:51.196 INFO:teuthology.orchestra.run.trial043.stdout:Build Date : Wed 03 Dec 2025 12:10:38 PM UTC 2026-02-26T02:36:51.196 INFO:teuthology.orchestra.run.trial043.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-26T02:36:51.196 INFO:teuthology.orchestra.run.trial043.stdout:Packager : builder@centos.org 2026-02-26T02:36:51.196 INFO:teuthology.orchestra.run.trial043.stdout:Vendor : CentOS 2026-02-26T02:36:51.196 INFO:teuthology.orchestra.run.trial043.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-26T02:36:51.196 INFO:teuthology.orchestra.run.trial043.stdout:Summary : Support tools for GRUB. 2026-02-26T02:36:51.196 INFO:teuthology.orchestra.run.trial043.stdout:Description : 2026-02-26T02:36:51.196 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:36:51.196 INFO:teuthology.orchestra.run.trial043.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-26T02:36:51.196 INFO:teuthology.orchestra.run.trial043.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-26T02:36:51.196 INFO:teuthology.orchestra.run.trial043.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-26T02:36:51.197 INFO:teuthology.orchestra.run.trial043.stdout:hardware devices. 2026-02-26T02:36:51.197 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:36:51.197 INFO:teuthology.orchestra.run.trial043.stdout:This subpackage provides tools for support of all platforms. 2026-02-26T02:36:51.198 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-26T02:36:51.198 INFO:teuthology.task.kernel:Updating grub on trial043 to boot 5.14.0-686.el9.x86_64 2026-02-26T02:36:51.198 DEBUG:teuthology.orchestra.run.trial043:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-26T02:36:51.262 INFO:teuthology.orchestra.run.trial043.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-26T02:36:51.263 DEBUG:teuthology.orchestra.run.trial043:> sudo command -v grubby && echo yes || echo no 2026-02-26T02:36:51.328 INFO:teuthology.orchestra.run.trial043.stdout:/sbin/grubby 2026-02-26T02:36:51.329 INFO:teuthology.orchestra.run.trial043.stdout:yes 2026-02-26T02:36:51.329 DEBUG:teuthology.orchestra.run.trial043:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-686.el9.x86_64" | head -n 1 2026-02-26T02:36:51.393 INFO:teuthology.orchestra.run.trial043.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64.conf 2026-02-26T02:36:51.395 DEBUG:teuthology.orchestra.run.trial043:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64 2026-02-26T02:36:51.468 DEBUG:teuthology.orchestra.run.trial043:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-26T02:36:51.675 INFO:teuthology.orchestra.run.trial043.stderr:Generating grub configuration file ... 2026-02-26T02:36:51.745 INFO:teuthology.orchestra.run.trial043.stderr:Found linux image: /boot/vmlinuz-5.14.0-686.el9.x86_64 2026-02-26T02:36:51.752 INFO:teuthology.orchestra.run.trial043.stderr:Found initrd image: /boot/initramfs-5.14.0-686.el9.x86_64.img 2026-02-26T02:36:52.001 INFO:teuthology.orchestra.run.trial043.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-26T02:36:52.008 INFO:teuthology.orchestra.run.trial043.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-26T02:36:52.042 INFO:teuthology.orchestra.run.trial043.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-26T02:36:52.048 INFO:teuthology.orchestra.run.trial043.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-26T02:36:52.080 INFO:teuthology.orchestra.run.trial043.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-26T02:36:52.086 INFO:teuthology.orchestra.run.trial043.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-26T02:36:52.109 INFO:teuthology.orchestra.run.trial043.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-26T02:36:52.115 INFO:teuthology.orchestra.run.trial043.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-26T02:36:52.202 INFO:teuthology.orchestra.run.trial043.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-26T02:36:52.211 INFO:teuthology.orchestra.run.trial043.stderr:done 2026-02-26T02:36:52.213 DEBUG:teuthology.orchestra.run.trial043:> test -d /sys/firmware/efi 2026-02-26T02:36:52.228 DEBUG:teuthology.orchestra.run.trial043:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-26T02:36:52.294 INFO:teuthology.orchestra.run.trial043.stdout:/boot/efi/EFI/BOOT 2026-02-26T02:36:52.294 INFO:teuthology.orchestra.run.trial043.stdout:/boot/efi/EFI/centos 2026-02-26T02:36:52.295 DEBUG:teuthology.orchestra.run.trial043:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-26T02:36:52.362 INFO:teuthology.orchestra.run.trial043.stdout:no 2026-02-26T02:36:52.363 DEBUG:teuthology.orchestra.run.trial043:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-26T02:36:52.426 INFO:teuthology.orchestra.run.trial043.stdout:yes 2026-02-26T02:36:52.426 DEBUG:teuthology.orchestra.run.trial043:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-26T02:36:52.490 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial043 2026-02-26T02:36:52.490 DEBUG:teuthology.orchestra.run.trial043:> sudo bash -c ' 2026-02-26T02:36:52.490 DEBUG:teuthology.orchestra.run.trial043:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-26T02:36:52.490 DEBUG:teuthology.orchestra.run.trial043:> ' 2026-02-26T02:37:22.533 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2026-02-26T02:37:22.534 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial043.front.sepia.ceph.com' 2026-02-26T02:37:22.535 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T02:37:36.833 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 10.20.193.43 2026-02-26T02:37:45.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial043.front.sepia.ceph.com' 2026-02-26T02:37:45.842 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T02:38:21.701 DEBUG:teuthology.orchestra.run.trial043:> true 2026-02-26T02:38:21.861 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial043.front.sepia.ceph.com' 2026-02-26T02:38:21.861 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-686.el9.x86_64"... 2026-02-26T02:38:21.861 DEBUG:teuthology.orchestra.run.trial043:> uname -r 2026-02-26T02:38:21.918 INFO:teuthology.orchestra.run.trial043.stdout:5.14.0-686.el9.x86_64 2026-02-26T02:38:21.918 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-686.el9.x86_64 vs 5.14.0-686.el9.x86_64 2026-02-26T02:38:21.918 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-26T02:38:21.918 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-26T02:38:22.919 INFO:teuthology.task.kernel:Enabling kdb on client.0... 2026-02-26T02:38:22.919 DEBUG:teuthology.orchestra.run.trial043:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-26T02:38:22.949 INFO:teuthology.orchestra.run.trial043.stdout:ttyS1 2026-02-26T02:38:22.971 DEBUG:teuthology.parallel:result is None 2026-02-26T02:38:22.971 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-26T02:38:22.979 INFO:teuthology.task.internal:Creating test directory... 2026-02-26T02:38:22.980 DEBUG:teuthology.orchestra.run.trial043:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-26T02:38:23.027 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-26T02:38:23.039 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-26T02:38:23.052 INFO:teuthology.task.internal:Creating archive directory... 2026-02-26T02:38:23.052 DEBUG:teuthology.orchestra.run.trial043:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-26T02:38:23.090 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-26T02:38:23.104 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-26T02:38:23.104 DEBUG:teuthology.orchestra.run.trial043:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-26T02:38:23.148 INFO:teuthology.orchestra.run.trial043.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-26T02:38:23.157 INFO:teuthology.orchestra.run.trial043.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-26T02:38:23.158 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-26T02:38:23.170 INFO:teuthology.task.internal:Configuring sudo... 2026-02-26T02:38:23.170 DEBUG:teuthology.orchestra.run.trial043:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-26T02:38:23.221 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-26T02:38:23.239 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-26T02:38:23.239 DEBUG:teuthology.orchestra.run.trial043:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-26T02:38:23.278 DEBUG:teuthology.orchestra.run.trial043:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-26T02:38:23.440 DEBUG:teuthology.orchestra.run.trial043:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-26T02:38:23.467 DEBUG:teuthology.orchestra.run.trial043:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-26T02:38:23.622 DEBUG:teuthology.orchestra.run.trial043:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-26T02:38:23.646 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-26T02:38:23.647 DEBUG:teuthology.orchestra.run.trial043:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-26T02:38:23.710 DEBUG:teuthology.orchestra.run.trial043:> sudo service rsyslog restart 2026-02-26T02:38:23.777 INFO:teuthology.orchestra.run.trial043.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-26T02:38:24.008 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-26T02:38:24.025 INFO:teuthology.task.internal:Starting timer... 2026-02-26T02:38:24.025 INFO:teuthology.run_tasks:Running task pcp... 2026-02-26T02:38:24.047 INFO:teuthology.run_tasks:Running task selinux... 2026-02-26T02:38:24.065 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2026-02-26T02:38:24.066 DEBUG:teuthology.orchestra.run.trial043:> sudo service auditd rotate 2026-02-26T02:38:24.094 INFO:teuthology.orchestra.run.trial043.stdout:Rotating logs: 2026-02-26T02:38:24.096 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-26T02:38:24.096 DEBUG:teuthology.orchestra.run.trial043:> /usr/sbin/getenforce 2026-02-26T02:38:24.151 INFO:teuthology.orchestra.run.trial043.stdout:Permissive 2026-02-26T02:38:24.151 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial043.front.sepia.ceph.com': 'permissive'} 2026-02-26T02:38:24.151 DEBUG:teuthology.orchestra.run.trial043:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2026-02-26T02:38:24.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T02:38:24.222 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-26T02:38:24.223 DEBUG:teuthology.orchestra.run.trial043:> sudo /usr/sbin/setenforce permissive 2026-02-26T02:38:24.285 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-26T02:38:24.307 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2026-02-26T02:38:24.310 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2026-02-26T02:38:24.420 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-26T02:38:24.428 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-26T02:38:24.429 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial043.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-26T02:41:11.606 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial043.front.sepia.ceph.com')] 2026-02-26T02:41:11.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial043.front.sepia.ceph.com' 2026-02-26T02:41:11.608 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T02:41:11.685 DEBUG:teuthology.orchestra.run.trial043:> true 2026-02-26T02:41:11.759 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial043.front.sepia.ceph.com' 2026-02-26T02:41:11.760 INFO:teuthology.run_tasks:Running task clock... 2026-02-26T02:41:11.777 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-26T02:41:11.777 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-26T02:41:11.778 DEBUG:teuthology.orchestra.run.trial043:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-26T02:41:11.827 INFO:teuthology.orchestra.run.trial043.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-26T02:41:11.839 INFO:teuthology.orchestra.run.trial043.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-26T02:41:11.861 INFO:teuthology.orchestra.run.trial043.stderr:sudo: ntpd: command not found 2026-02-26T02:41:11.870 INFO:teuthology.orchestra.run.trial043.stdout:506 Cannot talk to daemon 2026-02-26T02:41:11.883 INFO:teuthology.orchestra.run.trial043.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-26T02:41:11.894 INFO:teuthology.orchestra.run.trial043.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-26T02:41:11.943 INFO:teuthology.orchestra.run.trial043.stderr:bash: line 1: ntpq: command not found 2026-02-26T02:41:11.970 INFO:teuthology.orchestra.run.trial043.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-26T02:41:11.970 INFO:teuthology.orchestra.run.trial043.stdout:=============================================================================== 2026-02-26T02:41:11.970 INFO:teuthology.orchestra.run.trial043.stdout:^? LAX.CALTICK.NET 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-26T02:41:11.970 INFO:teuthology.orchestra.run.trial043.stdout:^? ntp.maxhost.io 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-26T02:41:11.970 INFO:teuthology.orchestra.run.trial043.stdout:^? ntp.maxhost.io 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-26T02:41:11.970 INFO:teuthology.orchestra.run.trial043.stdout:^? 155.248.196.28 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-26T02:41:11.972 INFO:teuthology.run_tasks:Running task install... 2026-02-26T02:41:11.986 DEBUG:teuthology.task.install:project ceph 2026-02-26T02:41:11.986 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5'}} 2026-02-26T02:41:11.986 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5'} 2026-02-26T02:41:11.986 INFO:teuthology.task.install:Using flavor: default 2026-02-26T02:41:11.997 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-26T02:41:11.997 INFO:teuthology.task.install:extra packages: [] 2026-02-26T02:41:11.997 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5', 'tag': None, 'wait_for_package': False} 2026-02-26T02:41:11.998 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5 2026-02-26T02:41:12.167 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri9-testing-2026-02-25-1600-squid/8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5/centos/9/flavors/default/ 2026-02-26T02:41:12.167 INFO:teuthology.task.install.rpm:Package version is 19.2.3-687.g8d724284 2026-02-26T02:41:12.306 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri9-testing-2026-02-25-1600-squid/8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri9-testing-2026-02-25-1600-squid/8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri9-testing-2026-02-25-1600-squid/8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-26T02:41:12.306 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-26T02:41:12.306 DEBUG:teuthology.orchestra.run.trial043:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-26T02:41:12.331 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-26T02:41:12.331 DEBUG:teuthology.orchestra.run.trial043:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-26T02:41:12.397 DEBUG:teuthology.orchestra.run.trial043:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-26T02:41:12.471 DEBUG:teuthology.orchestra.run.trial043:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-26T02:41:12.534 INFO:teuthology.orchestra.run.trial043.stdout:check_obsoletes = 1 2026-02-26T02:41:12.536 DEBUG:teuthology.orchestra.run.trial043:> sudo yum clean all 2026-02-26T02:41:12.684 INFO:teuthology.orchestra.run.trial043.stdout:41 files removed 2026-02-26T02:41:12.699 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-radosgw 2026-02-26T02:41:12.982 INFO:teuthology.orchestra.run.trial043.stdout:ceph packages for x86_64 651 kB/s | 86 kB 00:00 2026-02-26T02:41:13.120 INFO:teuthology.orchestra.run.trial043.stdout:ceph noarch packages 108 kB/s | 12 kB 00:00 2026-02-26T02:41:13.249 INFO:teuthology.orchestra.run.trial043.stdout:ceph source packages 18 kB/s | 1.9 kB 00:00 2026-02-26T02:41:17.097 INFO:teuthology.orchestra.run.trial043.stdout:CentOS Stream 9 - BaseOS 2.3 MB/s | 8.9 MB 00:03 2026-02-26T02:41:20.360 INFO:teuthology.orchestra.run.trial043.stdout:CentOS Stream 9 - AppStream 9.9 MB/s | 27 MB 00:02 2026-02-26T02:41:28.233 INFO:teuthology.orchestra.run.trial043.stdout:CentOS Stream 9 - CRB 1.5 MB/s | 8.0 MB 00:05 2026-02-26T02:41:29.863 INFO:teuthology.orchestra.run.trial043.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 20 kB 00:00 2026-02-26T02:41:30.862 INFO:teuthology.orchestra.run.trial043.stdout:Extra Packages for Enterprise Linux 22 MB/s | 20 MB 00:00 2026-02-26T02:41:34.020 INFO:teuthology.orchestra.run.trial043.stdout:lab-extras 2.7 MB/s | 50 kB 00:00 2026-02-26T02:41:35.303 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-26T02:41:35.305 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-26T02:41:35.305 INFO:teuthology.orchestra.run.trial043.stdout: Package Arch Version Repository Size 2026-02-26T02:41:35.305 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-26T02:41:35.305 INFO:teuthology.orchestra.run.trial043.stdout:Installing: 2026-02-26T02:41:35.305 INFO:teuthology.orchestra.run.trial043.stdout: ceph-radosgw x86_64 2:19.2.3-687.g8d724284.el9 ceph 11 M 2026-02-26T02:41:35.305 INFO:teuthology.orchestra.run.trial043.stdout:Upgrading: 2026-02-26T02:41:35.305 INFO:teuthology.orchestra.run.trial043.stdout: librados2 x86_64 2:19.2.3-687.g8d724284.el9 ceph 3.4 M 2026-02-26T02:41:35.305 INFO:teuthology.orchestra.run.trial043.stdout: librbd1 x86_64 2:19.2.3-687.g8d724284.el9 ceph 3.2 M 2026-02-26T02:41:35.305 INFO:teuthology.orchestra.run.trial043.stdout:Installing dependencies: 2026-02-26T02:41:35.305 INFO:teuthology.orchestra.run.trial043.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-26T02:41:35.305 INFO:teuthology.orchestra.run.trial043.stdout: ceph-base x86_64 2:19.2.3-687.g8d724284.el9 ceph 5.5 M 2026-02-26T02:41:35.305 INFO:teuthology.orchestra.run.trial043.stdout: ceph-common x86_64 2:19.2.3-687.g8d724284.el9 ceph 22 M 2026-02-26T02:41:35.305 INFO:teuthology.orchestra.run.trial043.stdout: ceph-selinux x86_64 2:19.2.3-687.g8d724284.el9 ceph 25 k 2026-02-26T02:41:35.305 INFO:teuthology.orchestra.run.trial043.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-26T02:41:35.306 INFO:teuthology.orchestra.run.trial043.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-26T02:41:35.306 INFO:teuthology.orchestra.run.trial043.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-26T02:41:35.306 INFO:teuthology.orchestra.run.trial043.stdout: libcephfs2 x86_64 2:19.2.3-687.g8d724284.el9 ceph 999 k 2026-02-26T02:41:35.306 INFO:teuthology.orchestra.run.trial043.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-26T02:41:35.306 INFO:teuthology.orchestra.run.trial043.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-26T02:41:35.306 INFO:teuthology.orchestra.run.trial043.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-26T02:41:35.306 INFO:teuthology.orchestra.run.trial043.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-26T02:41:35.306 INFO:teuthology.orchestra.run.trial043.stdout: libradosstriper1 x86_64 2:19.2.3-687.g8d724284.el9 ceph 504 k 2026-02-26T02:41:35.306 INFO:teuthology.orchestra.run.trial043.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-26T02:41:35.306 INFO:teuthology.orchestra.run.trial043.stdout: librgw2 x86_64 2:19.2.3-687.g8d724284.el9 ceph 5.4 M 2026-02-26T02:41:35.306 INFO:teuthology.orchestra.run.trial043.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-26T02:41:35.306 INFO:teuthology.orchestra.run.trial043.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-26T02:41:35.306 INFO:teuthology.orchestra.run.trial043.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-26T02:41:35.306 INFO:teuthology.orchestra.run.trial043.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-26T02:41:35.306 INFO:teuthology.orchestra.run.trial043.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-26T02:41:35.306 INFO:teuthology.orchestra.run.trial043.stdout: python3-ceph-argparse x86_64 2:19.2.3-687.g8d724284.el9 ceph 45 k 2026-02-26T02:41:35.306 INFO:teuthology.orchestra.run.trial043.stdout: python3-ceph-common x86_64 2:19.2.3-687.g8d724284.el9 ceph 142 k 2026-02-26T02:41:35.306 INFO:teuthology.orchestra.run.trial043.stdout: python3-cephfs x86_64 2:19.2.3-687.g8d724284.el9 ceph 165 k 2026-02-26T02:41:35.306 INFO:teuthology.orchestra.run.trial043.stdout: python3-rados x86_64 2:19.2.3-687.g8d724284.el9 ceph 321 k 2026-02-26T02:41:35.306 INFO:teuthology.orchestra.run.trial043.stdout: python3-rbd x86_64 2:19.2.3-687.g8d724284.el9 ceph 303 k 2026-02-26T02:41:35.306 INFO:teuthology.orchestra.run.trial043.stdout: python3-rgw x86_64 2:19.2.3-687.g8d724284.el9 ceph 100 k 2026-02-26T02:41:35.306 INFO:teuthology.orchestra.run.trial043.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-26T02:41:35.306 INFO:teuthology.orchestra.run.trial043.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-26T02:41:35.307 INFO:teuthology.orchestra.run.trial043.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-26T02:41:35.307 INFO:teuthology.orchestra.run.trial043.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-26T02:41:35.307 INFO:teuthology.orchestra.run.trial043.stdout:Installing weak dependencies: 2026-02-26T02:41:35.307 INFO:teuthology.orchestra.run.trial043.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-26T02:41:35.307 INFO:teuthology.orchestra.run.trial043.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-26T02:41:35.307 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:41:35.307 INFO:teuthology.orchestra.run.trial043.stdout:Transaction Summary 2026-02-26T02:41:35.307 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-26T02:41:35.307 INFO:teuthology.orchestra.run.trial043.stdout:Install 33 Packages 2026-02-26T02:41:35.307 INFO:teuthology.orchestra.run.trial043.stdout:Upgrade 2 Packages 2026-02-26T02:41:35.307 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:41:35.307 INFO:teuthology.orchestra.run.trial043.stdout:Total download size: 63 M 2026-02-26T02:41:35.307 INFO:teuthology.orchestra.run.trial043.stdout:Downloading Packages: 2026-02-26T02:41:37.803 INFO:teuthology.orchestra.run.trial043.stdout:(1/35): ceph-common-19.2.3-687.g8d724284.el9.x8 9.5 MB/s | 22 MB 00:02 2026-02-26T02:41:37.837 INFO:teuthology.orchestra.run.trial043.stdout:(2/35): ceph-selinux-19.2.3-687.g8d724284.el9.x 763 kB/s | 25 kB 00:00 2026-02-26T02:41:37.917 INFO:teuthology.orchestra.run.trial043.stdout:(3/35): libcephfs2-19.2.3-687.g8d724284.el9.x86 12 MB/s | 999 kB 00:00 2026-02-26T02:41:37.957 INFO:teuthology.orchestra.run.trial043.stdout:(4/35): libradosstriper1-19.2.3-687.g8d724284.e 12 MB/s | 504 kB 00:00 2026-02-26T02:41:38.234 INFO:teuthology.orchestra.run.trial043.stdout:(5/35): librgw2-19.2.3-687.g8d724284.el9.x86_64 19 MB/s | 5.4 MB 00:00 2026-02-26T02:41:38.252 INFO:teuthology.orchestra.run.trial043.stdout:(6/35): python3-ceph-argparse-19.2.3-687.g8d724 2.4 MB/s | 45 kB 00:00 2026-02-26T02:41:38.273 INFO:teuthology.orchestra.run.trial043.stdout:(7/35): python3-ceph-common-19.2.3-687.g8d72428 6.9 MB/s | 142 kB 00:00 2026-02-26T02:41:38.293 INFO:teuthology.orchestra.run.trial043.stdout:(8/35): python3-cephfs-19.2.3-687.g8d724284.el9 7.9 MB/s | 165 kB 00:00 2026-02-26T02:41:38.317 INFO:teuthology.orchestra.run.trial043.stdout:(9/35): python3-rados-19.2.3-687.g8d724284.el9. 13 MB/s | 321 kB 00:00 2026-02-26T02:41:38.340 INFO:teuthology.orchestra.run.trial043.stdout:(10/35): python3-rbd-19.2.3-687.g8d724284.el9.x 13 MB/s | 303 kB 00:00 2026-02-26T02:41:38.361 INFO:teuthology.orchestra.run.trial043.stdout:(11/35): python3-rgw-19.2.3-687.g8d724284.el9.x 4.7 MB/s | 100 kB 00:00 2026-02-26T02:41:38.377 INFO:teuthology.orchestra.run.trial043.stdout:(12/35): mailcap-2.1.49-5.el9.noarch.rpm 2.1 MB/s | 33 kB 00:00 2026-02-26T02:41:38.397 INFO:teuthology.orchestra.run.trial043.stdout:(13/35): nvme-cli-2.16-1.el9.x86_64.rpm 59 MB/s | 1.2 MB 00:00 2026-02-26T02:41:38.402 INFO:teuthology.orchestra.run.trial043.stdout:(14/35): pciutils-3.7.0-7.el9.x86_64.rpm 18 MB/s | 93 kB 00:00 2026-02-26T02:41:38.576 INFO:teuthology.orchestra.run.trial043.stdout:(15/35): boost-program-options-1.75.0-13.el9.x8 598 kB/s | 104 kB 00:00 2026-02-26T02:41:38.620 INFO:teuthology.orchestra.run.trial043.stdout:(16/35): libnbd-1.20.3-4.el9.x86_64.rpm 3.6 MB/s | 164 kB 00:00 2026-02-26T02:41:38.645 INFO:teuthology.orchestra.run.trial043.stdout:(17/35): libpmemobj-1.12.1-1.el9.x86_64.rpm 6.3 MB/s | 160 kB 00:00 2026-02-26T02:41:38.667 INFO:teuthology.orchestra.run.trial043.stdout:(18/35): librabbitmq-0.11.0-7.el9.x86_64.rpm 2.0 MB/s | 45 kB 00:00 2026-02-26T02:41:38.713 INFO:teuthology.orchestra.run.trial043.stdout:(19/35): librdkafka-1.6.1-102.el9.x86_64.rpm 14 MB/s | 662 kB 00:00 2026-02-26T02:41:38.738 INFO:teuthology.orchestra.run.trial043.stdout:(20/35): lttng-ust-2.12.0-6.el9.x86_64.rpm 12 MB/s | 292 kB 00:00 2026-02-26T02:41:38.755 INFO:teuthology.orchestra.run.trial043.stdout:(21/35): ceph-base-19.2.3-687.g8d724284.el9.x86 1.7 MB/s | 5.5 MB 00:03 2026-02-26T02:41:38.761 INFO:teuthology.orchestra.run.trial043.stdout:(22/35): qatlib-25.08.0-2.el9.x86_64.rpm 9.8 MB/s | 240 kB 00:00 2026-02-26T02:41:38.784 INFO:teuthology.orchestra.run.trial043.stdout:(23/35): qatzip-libs-1.3.1-1.el9.x86_64.rpm 2.9 MB/s | 66 kB 00:00 2026-02-26T02:41:38.810 INFO:teuthology.orchestra.run.trial043.stdout:(24/35): qatlib-service-25.08.0-2.el9.x86_64.rp 675 kB/s | 37 kB 00:00 2026-02-26T02:41:39.114 INFO:teuthology.orchestra.run.trial043.stdout:(25/35): gperftools-libs-2.9.1-3.el9.x86_64.rpm 933 kB/s | 308 kB 00:00 2026-02-26T02:41:39.166 INFO:teuthology.orchestra.run.trial043.stdout:(26/35): libarrow-doc-9.0.0-15.el9.noarch.rpm 475 kB/s | 25 kB 00:00 2026-02-26T02:41:39.214 INFO:teuthology.orchestra.run.trial043.stdout:(27/35): liboath-2.6.12-1.el9.x86_64.rpm 1.0 MB/s | 49 kB 00:00 2026-02-26T02:41:39.263 INFO:teuthology.orchestra.run.trial043.stdout:(28/35): libunwind-1.6.2-1.el9.x86_64.rpm 1.4 MB/s | 67 kB 00:00 2026-02-26T02:41:39.296 INFO:teuthology.orchestra.run.trial043.stdout:(29/35): libarrow-9.0.0-15.el9.x86_64.rpm 9.1 MB/s | 4.4 MB 00:00 2026-02-26T02:41:39.345 INFO:teuthology.orchestra.run.trial043.stdout:(30/35): re2-20211101-20.el9.x86_64.rpm 3.8 MB/s | 191 kB 00:00 2026-02-26T02:41:39.389 INFO:teuthology.orchestra.run.trial043.stdout:(31/35): parquet-libs-9.0.0-15.el9.x86_64.rpm 6.5 MB/s | 838 kB 00:00 2026-02-26T02:41:39.416 INFO:teuthology.orchestra.run.trial043.stdout:(32/35): ceph-radosgw-19.2.3-687.g8d724284.el9. 2.7 MB/s | 11 MB 00:03 2026-02-26T02:41:39.420 INFO:teuthology.orchestra.run.trial043.stdout:(33/35): thrift-0.15.0-4.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2026-02-26T02:41:39.596 INFO:teuthology.orchestra.run.trial043.stdout:(34/35): librbd1-19.2.3-687.g8d724284.el9.x86_6 18 MB/s | 3.2 MB 00:00 2026-02-26T02:41:39.988 INFO:teuthology.orchestra.run.trial043.stdout:(35/35): librados2-19.2.3-687.g8d724284.el9.x86 5.7 MB/s | 3.4 MB 00:00 2026-02-26T02:41:39.991 INFO:teuthology.orchestra.run.trial043.stdout:-------------------------------------------------------------------------------- 2026-02-26T02:41:39.991 INFO:teuthology.orchestra.run.trial043.stdout:Total 13 MB/s | 63 MB 00:04 2026-02-26T02:41:40.088 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction check 2026-02-26T02:41:40.106 INFO:teuthology.orchestra.run.trial043.stdout:Transaction check succeeded. 2026-02-26T02:41:40.106 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction test 2026-02-26T02:41:40.355 INFO:teuthology.orchestra.run.trial043.stdout:Transaction test succeeded. 2026-02-26T02:41:40.356 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction 2026-02-26T02:41:40.648 INFO:teuthology.orchestra.run.trial043.stdout: Preparing : 1/1 2026-02-26T02:41:40.656 INFO:teuthology.orchestra.run.trial043.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/37 2026-02-26T02:41:40.658 INFO:teuthology.orchestra.run.trial043.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/37 2026-02-26T02:41:40.670 INFO:teuthology.orchestra.run.trial043.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/37 2026-02-26T02:41:40.820 INFO:teuthology.orchestra.run.trial043.stdout: Installing : thrift-0.15.0-4.el9.x86_64 4/37 2026-02-26T02:41:40.824 INFO:teuthology.orchestra.run.trial043.stdout: Upgrading : librados2-2:19.2.3-687.g8d724284.el9.x86_64 5/37 2026-02-26T02:41:40.853 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: librados2-2:19.2.3-687.g8d724284.el9.x86_64 5/37 2026-02-26T02:41:40.896 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-rados-2:19.2.3-687.g8d724284.el9.x86_64 6/37 2026-02-26T02:41:40.897 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libcephfs2-2:19.2.3-687.g8d724284.el9.x86_64 7/37 2026-02-26T02:41:40.909 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: libcephfs2-2:19.2.3-687.g8d724284.el9.x86_64 7/37 2026-02-26T02:41:40.913 INFO:teuthology.orchestra.run.trial043.stdout: Installing : liboath-2.6.12-1.el9.x86_64 8/37 2026-02-26T02:41:40.922 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-ceph-argparse-2:19.2.3-687.g8d724284.el9.x 9/37 2026-02-26T02:41:40.945 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-cephfs-2:19.2.3-687.g8d724284.el9.x86_64 10/37 2026-02-26T02:41:40.945 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libradosstriper1-2:19.2.3-687.g8d724284.el9.x86_64 11/37 2026-02-26T02:41:40.958 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: libradosstriper1-2:19.2.3-687.g8d724284.el9.x86_64 11/37 2026-02-26T02:41:40.994 INFO:teuthology.orchestra.run.trial043.stdout: Installing : re2-1:20211101-20.el9.x86_64 12/37 2026-02-26T02:41:40.999 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 13/37 2026-02-26T02:41:41.005 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/37 2026-02-26T02:41:41.007 INFO:teuthology.orchestra.run.trial043.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/37 2026-02-26T02:41:41.010 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 16/37 2026-02-26T02:41:41.015 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/37 2026-02-26T02:41:41.156 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 18/37 2026-02-26T02:41:41.157 INFO:teuthology.orchestra.run.trial043.stdout: Upgrading : librbd1-2:19.2.3-687.g8d724284.el9.x86_64 19/37 2026-02-26T02:41:41.182 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: librbd1-2:19.2.3-687.g8d724284.el9.x86_64 19/37 2026-02-26T02:41:41.184 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-rbd-2:19.2.3-687.g8d724284.el9.x86_64 20/37 2026-02-26T02:41:41.191 INFO:teuthology.orchestra.run.trial043.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 21/37 2026-02-26T02:41:41.416 INFO:teuthology.orchestra.run.trial043.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 22/37 2026-02-26T02:41:41.417 INFO:teuthology.orchestra.run.trial043.stdout: Installing : librgw2-2:19.2.3-687.g8d724284.el9.x86_64 23/37 2026-02-26T02:41:41.432 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: librgw2-2:19.2.3-687.g8d724284.el9.x86_64 23/37 2026-02-26T02:41:41.437 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-rgw-2:19.2.3-687.g8d724284.el9.x86_64 24/37 2026-02-26T02:41:41.441 INFO:teuthology.orchestra.run.trial043.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 25/37 2026-02-26T02:41:41.443 INFO:teuthology.orchestra.run.trial043.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 26/37 2026-02-26T02:41:41.444 INFO:teuthology.orchestra.run.trial043.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 27/37 2026-02-26T02:41:41.461 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 27/37 2026-02-26T02:41:41.575 INFO:teuthology.orchestra.run.trial043.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 28/37 2026-02-26T02:41:41.577 INFO:teuthology.orchestra.run.trial043.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 29/37 2026-02-26T02:41:41.859 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 29/37 2026-02-26T02:41:41.859 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-26T02:41:41.859 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:41:41.895 INFO:teuthology.orchestra.run.trial043.stdout: Installing : mailcap-2.1.49-5.el9.noarch 30/37 2026-02-26T02:41:41.897 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-ceph-common-2:19.2.3-687.g8d724284.el9.x86 31/37 2026-02-26T02:41:42.877 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-common-2:19.2.3-687.g8d724284.el9.x86_64 32/37 2026-02-26T02:41:42.879 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-common-2:19.2.3-687.g8d724284.el9.x86_64 32/37 2026-02-26T02:41:43.164 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-common-2:19.2.3-687.g8d724284.el9.x86_64 32/37 2026-02-26T02:41:43.167 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-base-2:19.2.3-687.g8d724284.el9.x86_64 33/37 2026-02-26T02:41:43.210 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-base-2:19.2.3-687.g8d724284.el9.x86_64 33/37 2026-02-26T02:41:43.210 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-26T02:41:43.210 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-26T02:41:43.210 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:41:43.213 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-selinux-2:19.2.3-687.g8d724284.el9.x86_64 34/37 2026-02-26T02:41:47.703 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-selinux-2:19.2.3-687.g8d724284.el9.x86_64 34/37 2026-02-26T02:41:47.703 INFO:teuthology.orchestra.run.trial043.stdout:skipping the directory /sys 2026-02-26T02:41:47.703 INFO:teuthology.orchestra.run.trial043.stdout:skipping the directory /proc 2026-02-26T02:41:47.703 INFO:teuthology.orchestra.run.trial043.stdout:skipping the directory /mnt 2026-02-26T02:41:47.703 INFO:teuthology.orchestra.run.trial043.stdout:skipping the directory /var/tmp 2026-02-26T02:41:47.703 INFO:teuthology.orchestra.run.trial043.stdout:skipping the directory /home 2026-02-26T02:41:47.704 INFO:teuthology.orchestra.run.trial043.stdout:skipping the directory /root 2026-02-26T02:41:47.704 INFO:teuthology.orchestra.run.trial043.stdout:skipping the directory /tmp 2026-02-26T02:41:47.704 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:41:48.156 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-radosgw-2:19.2.3-687.g8d724284.el9.x86_64 35/37 2026-02-26T02:41:48.179 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-radosgw-2:19.2.3-687.g8d724284.el9.x86_64 35/37 2026-02-26T02:41:48.180 INFO:teuthology.orchestra.run.trial043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T02:41:48.180 INFO:teuthology.orchestra.run.trial043.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-26T02:41:48.180 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-26T02:41:48.180 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-26T02:41:48.180 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:41:48.182 INFO:teuthology.orchestra.run.trial043.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 36/37 2026-02-26T02:41:48.195 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 36/37 2026-02-26T02:41:48.195 INFO:teuthology.orchestra.run.trial043.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 37/37 2026-02-26T02:41:48.250 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 37/37 2026-02-26T02:41:48.250 INFO:teuthology.orchestra.run.trial043.stdout:Creating group 'qat' with GID 992. 2026-02-26T02:41:48.250 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:41:48.542 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-base-2:19.2.3-687.g8d724284.el9.x86_64 1/37 2026-02-26T02:41:48.543 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-common-2:19.2.3-687.g8d724284.el9.x86_64 2/37 2026-02-26T02:41:48.543 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-radosgw-2:19.2.3-687.g8d724284.el9.x86_64 3/37 2026-02-26T02:41:48.543 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-selinux-2:19.2.3-687.g8d724284.el9.x86_64 4/37 2026-02-26T02:41:48.543 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libcephfs2-2:19.2.3-687.g8d724284.el9.x86_64 5/37 2026-02-26T02:41:48.543 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libradosstriper1-2:19.2.3-687.g8d724284.el9.x86_64 6/37 2026-02-26T02:41:48.543 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : librgw2-2:19.2.3-687.g8d724284.el9.x86_64 7/37 2026-02-26T02:41:48.543 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-ceph-argparse-2:19.2.3-687.g8d724284.el9.x 8/37 2026-02-26T02:41:48.543 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-ceph-common-2:19.2.3-687.g8d724284.el9.x86 9/37 2026-02-26T02:41:48.543 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-cephfs-2:19.2.3-687.g8d724284.el9.x86_64 10/37 2026-02-26T02:41:48.543 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-rados-2:19.2.3-687.g8d724284.el9.x86_64 11/37 2026-02-26T02:41:48.543 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-rbd-2:19.2.3-687.g8d724284.el9.x86_64 12/37 2026-02-26T02:41:48.543 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-rgw-2:19.2.3-687.g8d724284.el9.x86_64 13/37 2026-02-26T02:41:48.544 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 14/37 2026-02-26T02:41:48.544 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 15/37 2026-02-26T02:41:48.544 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 16/37 2026-02-26T02:41:48.544 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 17/37 2026-02-26T02:41:48.544 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 18/37 2026-02-26T02:41:48.544 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 19/37 2026-02-26T02:41:48.544 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 20/37 2026-02-26T02:41:48.544 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 21/37 2026-02-26T02:41:48.544 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 22/37 2026-02-26T02:41:48.544 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 23/37 2026-02-26T02:41:48.544 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 24/37 2026-02-26T02:41:48.544 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 25/37 2026-02-26T02:41:48.545 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 26/37 2026-02-26T02:41:48.545 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 27/37 2026-02-26T02:41:48.545 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 28/37 2026-02-26T02:41:48.545 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 29/37 2026-02-26T02:41:48.545 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 30/37 2026-02-26T02:41:48.545 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 31/37 2026-02-26T02:41:48.545 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : re2-1:20211101-20.el9.x86_64 32/37 2026-02-26T02:41:48.545 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 33/37 2026-02-26T02:41:48.546 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : librados2-2:19.2.3-687.g8d724284.el9.x86_64 34/37 2026-02-26T02:41:48.546 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 35/37 2026-02-26T02:41:48.546 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : librbd1-2:19.2.3-687.g8d724284.el9.x86_64 36/37 2026-02-26T02:41:48.610 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 37/37 2026-02-26T02:41:48.610 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:41:48.610 INFO:teuthology.orchestra.run.trial043.stdout:Upgraded: 2026-02-26T02:41:48.610 INFO:teuthology.orchestra.run.trial043.stdout: librados2-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T02:41:48.611 INFO:teuthology.orchestra.run.trial043.stdout: librbd1-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T02:41:48.611 INFO:teuthology.orchestra.run.trial043.stdout:Installed: 2026-02-26T02:41:48.611 INFO:teuthology.orchestra.run.trial043.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-26T02:41:48.611 INFO:teuthology.orchestra.run.trial043.stdout: ceph-base-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T02:41:48.611 INFO:teuthology.orchestra.run.trial043.stdout: ceph-common-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T02:41:48.611 INFO:teuthology.orchestra.run.trial043.stdout: ceph-radosgw-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T02:41:48.611 INFO:teuthology.orchestra.run.trial043.stdout: ceph-selinux-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T02:41:48.611 INFO:teuthology.orchestra.run.trial043.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-26T02:41:48.611 INFO:teuthology.orchestra.run.trial043.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-26T02:41:48.611 INFO:teuthology.orchestra.run.trial043.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-26T02:41:48.611 INFO:teuthology.orchestra.run.trial043.stdout: libcephfs2-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T02:41:48.612 INFO:teuthology.orchestra.run.trial043.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-26T02:41:48.612 INFO:teuthology.orchestra.run.trial043.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-26T02:41:48.612 INFO:teuthology.orchestra.run.trial043.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-26T02:41:48.612 INFO:teuthology.orchestra.run.trial043.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-26T02:41:48.612 INFO:teuthology.orchestra.run.trial043.stdout: libradosstriper1-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T02:41:48.612 INFO:teuthology.orchestra.run.trial043.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-26T02:41:48.612 INFO:teuthology.orchestra.run.trial043.stdout: librgw2-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T02:41:48.612 INFO:teuthology.orchestra.run.trial043.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-26T02:41:48.612 INFO:teuthology.orchestra.run.trial043.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-26T02:41:48.612 INFO:teuthology.orchestra.run.trial043.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-26T02:41:48.612 INFO:teuthology.orchestra.run.trial043.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-26T02:41:48.612 INFO:teuthology.orchestra.run.trial043.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-26T02:41:48.612 INFO:teuthology.orchestra.run.trial043.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-26T02:41:48.613 INFO:teuthology.orchestra.run.trial043.stdout: python3-ceph-argparse-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T02:41:48.613 INFO:teuthology.orchestra.run.trial043.stdout: python3-ceph-common-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T02:41:48.613 INFO:teuthology.orchestra.run.trial043.stdout: python3-cephfs-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T02:41:48.613 INFO:teuthology.orchestra.run.trial043.stdout: python3-rados-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T02:41:48.613 INFO:teuthology.orchestra.run.trial043.stdout: python3-rbd-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T02:41:48.613 INFO:teuthology.orchestra.run.trial043.stdout: python3-rgw-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T02:41:48.613 INFO:teuthology.orchestra.run.trial043.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-26T02:41:48.613 INFO:teuthology.orchestra.run.trial043.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-26T02:41:48.613 INFO:teuthology.orchestra.run.trial043.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-26T02:41:48.613 INFO:teuthology.orchestra.run.trial043.stdout: re2-1:20211101-20.el9.x86_64 2026-02-26T02:41:48.613 INFO:teuthology.orchestra.run.trial043.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-26T02:41:48.613 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:41:48.614 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-26T02:41:48.651 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-test 2026-02-26T02:41:48.967 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:14 ago on Thu 26 Feb 2026 02:41:34 AM UTC. 2026-02-26T02:41:49.051 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-26T02:41:49.051 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-26T02:41:49.051 INFO:teuthology.orchestra.run.trial043.stdout: Package Arch Version Repository Size 2026-02-26T02:41:49.051 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-26T02:41:49.051 INFO:teuthology.orchestra.run.trial043.stdout:Installing: 2026-02-26T02:41:49.051 INFO:teuthology.orchestra.run.trial043.stdout: ceph-test x86_64 2:19.2.3-687.g8d724284.el9 ceph 51 M 2026-02-26T02:41:49.052 INFO:teuthology.orchestra.run.trial043.stdout:Installing dependencies: 2026-02-26T02:41:49.052 INFO:teuthology.orchestra.run.trial043.stdout: libcephsqlite x86_64 2:19.2.3-687.g8d724284.el9 ceph 163 k 2026-02-26T02:41:49.052 INFO:teuthology.orchestra.run.trial043.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-26T02:41:49.052 INFO:teuthology.orchestra.run.trial043.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-26T02:41:49.052 INFO:teuthology.orchestra.run.trial043.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-26T02:41:49.052 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:41:49.052 INFO:teuthology.orchestra.run.trial043.stdout:Transaction Summary 2026-02-26T02:41:49.052 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-26T02:41:49.052 INFO:teuthology.orchestra.run.trial043.stdout:Install 5 Packages 2026-02-26T02:41:49.052 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:41:49.052 INFO:teuthology.orchestra.run.trial043.stdout:Total download size: 51 M 2026-02-26T02:41:49.052 INFO:teuthology.orchestra.run.trial043.stdout:Installed size: 222 M 2026-02-26T02:41:49.052 INFO:teuthology.orchestra.run.trial043.stdout:Downloading Packages: 2026-02-26T02:41:49.558 INFO:teuthology.orchestra.run.trial043.stdout:(1/5): libcephsqlite-19.2.3-687.g8d724284.el9.x 388 kB/s | 163 kB 00:00 2026-02-26T02:41:49.635 INFO:teuthology.orchestra.run.trial043.stdout:(2/5): libxslt-1.1.34-12.el9.x86_64.rpm 468 kB/s | 233 kB 00:00 2026-02-26T02:41:49.728 INFO:teuthology.orchestra.run.trial043.stdout:(3/5): xmlstarlet-1.6.1-20.el9.x86_64.rpm 684 kB/s | 64 kB 00:00 2026-02-26T02:41:50.081 INFO:teuthology.orchestra.run.trial043.stdout:(4/5): socat-1.7.4.1-8.el9.x86_64.rpm 580 kB/s | 303 kB 00:00 2026-02-26T02:41:53.464 INFO:teuthology.orchestra.run.trial043.stdout:(5/5): ceph-test-19.2.3-687.g8d724284.el9.x86_6 12 MB/s | 51 MB 00:04 2026-02-26T02:41:53.465 INFO:teuthology.orchestra.run.trial043.stdout:-------------------------------------------------------------------------------- 2026-02-26T02:41:53.465 INFO:teuthology.orchestra.run.trial043.stdout:Total 12 MB/s | 51 MB 00:04 2026-02-26T02:41:53.491 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction check 2026-02-26T02:41:53.499 INFO:teuthology.orchestra.run.trial043.stdout:Transaction check succeeded. 2026-02-26T02:41:53.500 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction test 2026-02-26T02:41:53.689 INFO:teuthology.orchestra.run.trial043.stdout:Transaction test succeeded. 2026-02-26T02:41:53.690 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction 2026-02-26T02:41:53.906 INFO:teuthology.orchestra.run.trial043.stdout: Preparing : 1/1 2026-02-26T02:41:53.912 INFO:teuthology.orchestra.run.trial043.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 1/5 2026-02-26T02:41:53.916 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 2/5 2026-02-26T02:41:53.924 INFO:teuthology.orchestra.run.trial043.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/5 2026-02-26T02:41:53.925 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libcephsqlite-2:19.2.3-687.g8d724284.el9.x86_64 4/5 2026-02-26T02:41:56.332 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: libcephsqlite-2:19.2.3-687.g8d724284.el9.x86_64 4/5 2026-02-26T02:41:56.351 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-test-2:19.2.3-687.g8d724284.el9.x86_64 5/5 2026-02-26T02:41:56.606 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-test-2:19.2.3-687.g8d724284.el9.x86_64 5/5 2026-02-26T02:41:56.606 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-test-2:19.2.3-687.g8d724284.el9.x86_64 1/5 2026-02-26T02:41:56.607 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libcephsqlite-2:19.2.3-687.g8d724284.el9.x86_64 2/5 2026-02-26T02:41:56.607 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 3/5 2026-02-26T02:41:56.607 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 4/5 2026-02-26T02:41:56.666 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/5 2026-02-26T02:41:56.666 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:41:56.666 INFO:teuthology.orchestra.run.trial043.stdout:Installed: 2026-02-26T02:41:56.666 INFO:teuthology.orchestra.run.trial043.stdout: ceph-test-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T02:41:56.666 INFO:teuthology.orchestra.run.trial043.stdout: libcephsqlite-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T02:41:56.666 INFO:teuthology.orchestra.run.trial043.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-26T02:41:56.666 INFO:teuthology.orchestra.run.trial043.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-26T02:41:56.666 INFO:teuthology.orchestra.run.trial043.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-26T02:41:56.666 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:41:56.666 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-26T02:41:56.698 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph 2026-02-26T02:41:57.017 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:23 ago on Thu 26 Feb 2026 02:41:34 AM UTC. 2026-02-26T02:41:57.106 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-26T02:41:57.108 INFO:teuthology.orchestra.run.trial043.stdout:====================================================================================== 2026-02-26T02:41:57.109 INFO:teuthology.orchestra.run.trial043.stdout: Package Arch Version Repository Size 2026-02-26T02:41:57.109 INFO:teuthology.orchestra.run.trial043.stdout:====================================================================================== 2026-02-26T02:41:57.109 INFO:teuthology.orchestra.run.trial043.stdout:Installing: 2026-02-26T02:41:57.109 INFO:teuthology.orchestra.run.trial043.stdout: ceph x86_64 2:19.2.3-687.g8d724284.el9 ceph 6.5 k 2026-02-26T02:41:57.109 INFO:teuthology.orchestra.run.trial043.stdout:Installing dependencies: 2026-02-26T02:41:57.109 INFO:teuthology.orchestra.run.trial043.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-26T02:41:57.109 INFO:teuthology.orchestra.run.trial043.stdout: ceph-grafana-dashboards noarch 2:19.2.3-687.g8d724284.el9 ceph-noarch 31 k 2026-02-26T02:41:57.109 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mds x86_64 2:19.2.3-687.g8d724284.el9 ceph 2.4 M 2026-02-26T02:41:57.110 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mgr x86_64 2:19.2.3-687.g8d724284.el9 ceph 1.1 M 2026-02-26T02:41:57.110 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mgr-modules-core noarch 2:19.2.3-687.g8d724284.el9 ceph-noarch 253 k 2026-02-26T02:41:57.110 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mon x86_64 2:19.2.3-687.g8d724284.el9 ceph 4.7 M 2026-02-26T02:41:57.110 INFO:teuthology.orchestra.run.trial043.stdout: ceph-osd x86_64 2:19.2.3-687.g8d724284.el9 ceph 17 M 2026-02-26T02:41:57.110 INFO:teuthology.orchestra.run.trial043.stdout: ceph-prometheus-alerts noarch 2:19.2.3-687.g8d724284.el9 ceph-noarch 17 k 2026-02-26T02:41:57.110 INFO:teuthology.orchestra.run.trial043.stdout: cephadm noarch 2:19.2.3-687.g8d724284.el9 ceph-noarch 769 k 2026-02-26T02:41:57.110 INFO:teuthology.orchestra.run.trial043.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-26T02:41:57.110 INFO:teuthology.orchestra.run.trial043.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-26T02:41:57.110 INFO:teuthology.orchestra.run.trial043.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-26T02:41:57.110 INFO:teuthology.orchestra.run.trial043.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-26T02:41:57.110 INFO:teuthology.orchestra.run.trial043.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-26T02:41:57.111 INFO:teuthology.orchestra.run.trial043.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-26T02:41:57.111 INFO:teuthology.orchestra.run.trial043.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-26T02:41:57.111 INFO:teuthology.orchestra.run.trial043.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-26T02:41:57.111 INFO:teuthology.orchestra.run.trial043.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-26T02:41:57.111 INFO:teuthology.orchestra.run.trial043.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-26T02:41:57.111 INFO:teuthology.orchestra.run.trial043.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-26T02:41:57.111 INFO:teuthology.orchestra.run.trial043.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-26T02:41:57.111 INFO:teuthology.orchestra.run.trial043.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-26T02:41:57.111 INFO:teuthology.orchestra.run.trial043.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-26T02:41:57.111 INFO:teuthology.orchestra.run.trial043.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-26T02:41:57.111 INFO:teuthology.orchestra.run.trial043.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-26T02:41:57.111 INFO:teuthology.orchestra.run.trial043.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-26T02:41:57.112 INFO:teuthology.orchestra.run.trial043.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-26T02:41:57.112 INFO:teuthology.orchestra.run.trial043.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-26T02:41:57.112 INFO:teuthology.orchestra.run.trial043.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-26T02:41:57.112 INFO:teuthology.orchestra.run.trial043.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-26T02:41:57.112 INFO:teuthology.orchestra.run.trial043.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-26T02:41:57.112 INFO:teuthology.orchestra.run.trial043.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-26T02:41:57.112 INFO:teuthology.orchestra.run.trial043.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-26T02:41:57.112 INFO:teuthology.orchestra.run.trial043.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-26T02:41:57.112 INFO:teuthology.orchestra.run.trial043.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-26T02:41:57.112 INFO:teuthology.orchestra.run.trial043.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-26T02:41:57.112 INFO:teuthology.orchestra.run.trial043.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-26T02:41:57.112 INFO:teuthology.orchestra.run.trial043.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-26T02:41:57.112 INFO:teuthology.orchestra.run.trial043.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-26T02:41:57.113 INFO:teuthology.orchestra.run.trial043.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-26T02:41:57.113 INFO:teuthology.orchestra.run.trial043.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-26T02:41:57.113 INFO:teuthology.orchestra.run.trial043.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-26T02:41:57.113 INFO:teuthology.orchestra.run.trial043.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-26T02:41:57.113 INFO:teuthology.orchestra.run.trial043.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-26T02:41:57.113 INFO:teuthology.orchestra.run.trial043.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-26T02:41:57.113 INFO:teuthology.orchestra.run.trial043.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-26T02:41:57.113 INFO:teuthology.orchestra.run.trial043.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-26T02:41:57.113 INFO:teuthology.orchestra.run.trial043.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-26T02:41:57.113 INFO:teuthology.orchestra.run.trial043.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-26T02:41:57.113 INFO:teuthology.orchestra.run.trial043.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-26T02:41:57.113 INFO:teuthology.orchestra.run.trial043.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-26T02:41:57.113 INFO:teuthology.orchestra.run.trial043.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2026-02-26T02:41:57.114 INFO:teuthology.orchestra.run.trial043.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-26T02:41:57.114 INFO:teuthology.orchestra.run.trial043.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2026-02-26T02:41:57.114 INFO:teuthology.orchestra.run.trial043.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-26T02:41:57.114 INFO:teuthology.orchestra.run.trial043.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-26T02:41:57.114 INFO:teuthology.orchestra.run.trial043.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-26T02:41:57.114 INFO:teuthology.orchestra.run.trial043.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-26T02:41:57.114 INFO:teuthology.orchestra.run.trial043.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-26T02:41:57.114 INFO:teuthology.orchestra.run.trial043.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-26T02:41:57.114 INFO:teuthology.orchestra.run.trial043.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-26T02:41:57.114 INFO:teuthology.orchestra.run.trial043.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2026-02-26T02:41:57.114 INFO:teuthology.orchestra.run.trial043.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-26T02:41:57.114 INFO:teuthology.orchestra.run.trial043.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-26T02:41:57.114 INFO:teuthology.orchestra.run.trial043.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-26T02:41:57.115 INFO:teuthology.orchestra.run.trial043.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-26T02:41:57.115 INFO:teuthology.orchestra.run.trial043.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-02-26T02:41:57.115 INFO:teuthology.orchestra.run.trial043.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-02-26T02:41:57.115 INFO:teuthology.orchestra.run.trial043.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-26T02:41:57.115 INFO:teuthology.orchestra.run.trial043.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-26T02:41:57.115 INFO:teuthology.orchestra.run.trial043.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-26T02:41:57.115 INFO:teuthology.orchestra.run.trial043.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-26T02:41:57.115 INFO:teuthology.orchestra.run.trial043.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-26T02:41:57.115 INFO:teuthology.orchestra.run.trial043.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-26T02:41:57.115 INFO:teuthology.orchestra.run.trial043.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-26T02:41:57.115 INFO:teuthology.orchestra.run.trial043.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-26T02:41:57.115 INFO:teuthology.orchestra.run.trial043.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-26T02:41:57.115 INFO:teuthology.orchestra.run.trial043.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-26T02:41:57.116 INFO:teuthology.orchestra.run.trial043.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-26T02:41:57.116 INFO:teuthology.orchestra.run.trial043.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-26T02:41:57.116 INFO:teuthology.orchestra.run.trial043.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2026-02-26T02:41:57.116 INFO:teuthology.orchestra.run.trial043.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-26T02:41:57.116 INFO:teuthology.orchestra.run.trial043.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2026-02-26T02:41:57.116 INFO:teuthology.orchestra.run.trial043.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-26T02:41:57.116 INFO:teuthology.orchestra.run.trial043.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-26T02:41:57.116 INFO:teuthology.orchestra.run.trial043.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-26T02:41:57.116 INFO:teuthology.orchestra.run.trial043.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-26T02:41:57.116 INFO:teuthology.orchestra.run.trial043.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-26T02:41:57.116 INFO:teuthology.orchestra.run.trial043.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-26T02:41:57.116 INFO:teuthology.orchestra.run.trial043.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-26T02:41:57.116 INFO:teuthology.orchestra.run.trial043.stdout:Installing weak dependencies: 2026-02-26T02:41:57.117 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mgr-cephadm noarch 2:19.2.3-687.g8d724284.el9 ceph-noarch 150 k 2026-02-26T02:41:57.117 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mgr-dashboard noarch 2:19.2.3-687.g8d724284.el9 ceph-noarch 3.8 M 2026-02-26T02:41:57.117 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mgr-diskprediction-local noarch 2:19.2.3-687.g8d724284.el9 ceph-noarch 7.4 M 2026-02-26T02:41:57.117 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mgr-k8sevents noarch 2:19.2.3-687.g8d724284.el9 ceph-noarch 22 k 2026-02-26T02:41:57.117 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mgr-rook noarch 2:19.2.3-687.g8d724284.el9 ceph-noarch 49 k 2026-02-26T02:41:57.117 INFO:teuthology.orchestra.run.trial043.stdout: ceph-volume noarch 2:19.2.3-687.g8d724284.el9 ceph-noarch 299 k 2026-02-26T02:41:57.117 INFO:teuthology.orchestra.run.trial043.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-26T02:41:57.117 INFO:teuthology.orchestra.run.trial043.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-26T02:41:57.117 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:41:57.117 INFO:teuthology.orchestra.run.trial043.stdout:Transaction Summary 2026-02-26T02:41:57.117 INFO:teuthology.orchestra.run.trial043.stdout:====================================================================================== 2026-02-26T02:41:57.117 INFO:teuthology.orchestra.run.trial043.stdout:Install 99 Packages 2026-02-26T02:41:57.118 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:41:57.118 INFO:teuthology.orchestra.run.trial043.stdout:Total download size: 95 M 2026-02-26T02:41:57.118 INFO:teuthology.orchestra.run.trial043.stdout:Installed size: 520 M 2026-02-26T02:41:57.119 INFO:teuthology.orchestra.run.trial043.stdout:Downloading Packages: 2026-02-26T02:41:58.062 INFO:teuthology.orchestra.run.trial043.stdout:(1/99): ceph-19.2.3-687.g8d724284.el9.x86_64.rp 58 kB/s | 6.5 kB 00:00 2026-02-26T02:41:58.376 INFO:teuthology.orchestra.run.trial043.stdout:(2/99): ceph-mgr-19.2.3-687.g8d724284.el9.x86_6 2.5 MB/s | 1.1 MB 00:00 2026-02-26T02:41:58.483 INFO:teuthology.orchestra.run.trial043.stdout:(3/99): ceph-mds-19.2.3-687.g8d724284.el9.x86_6 4.5 MB/s | 2.4 MB 00:00 2026-02-26T02:41:58.519 INFO:teuthology.orchestra.run.trial043.stdout:(4/99): ceph-grafana-dashboards-19.2.3-687.g8d7 872 kB/s | 31 kB 00:00 2026-02-26T02:41:58.557 INFO:teuthology.orchestra.run.trial043.stdout:(5/99): ceph-mgr-cephadm-19.2.3-687.g8d724284.e 3.8 MB/s | 150 kB 00:00 2026-02-26T02:41:58.968 INFO:teuthology.orchestra.run.trial043.stdout:(6/99): ceph-mgr-dashboard-19.2.3-687.g8d724284 9.3 MB/s | 3.8 MB 00:00 2026-02-26T02:41:59.052 INFO:teuthology.orchestra.run.trial043.stdout:(7/99): ceph-mon-19.2.3-687.g8d724284.el9.x86_6 4.8 MB/s | 4.7 MB 00:00 2026-02-26T02:41:59.099 INFO:teuthology.orchestra.run.trial043.stdout:(8/99): ceph-mgr-k8sevents-19.2.3-687.g8d724284 473 kB/s | 22 kB 00:00 2026-02-26T02:41:59.151 INFO:teuthology.orchestra.run.trial043.stdout:(9/99): ceph-mgr-modules-core-19.2.3-687.g8d724 4.7 MB/s | 253 kB 00:00 2026-02-26T02:41:59.201 INFO:teuthology.orchestra.run.trial043.stdout:(10/99): ceph-mgr-rook-19.2.3-687.g8d724284.el9 989 kB/s | 49 kB 00:00 2026-02-26T02:41:59.251 INFO:teuthology.orchestra.run.trial043.stdout:(11/99): ceph-prometheus-alerts-19.2.3-687.g8d7 335 kB/s | 17 kB 00:00 2026-02-26T02:41:59.308 INFO:teuthology.orchestra.run.trial043.stdout:(12/99): ceph-volume-19.2.3-687.g8d724284.el9.n 5.1 MB/s | 299 kB 00:00 2026-02-26T02:41:59.480 INFO:teuthology.orchestra.run.trial043.stdout:(13/99): cephadm-19.2.3-687.g8d724284.el9.noarc 4.4 MB/s | 769 kB 00:00 2026-02-26T02:41:59.558 INFO:teuthology.orchestra.run.trial043.stdout:(14/99): ceph-mgr-diskprediction-local-19.2.3-6 13 MB/s | 7.4 MB 00:00 2026-02-26T02:41:59.645 INFO:teuthology.orchestra.run.trial043.stdout:(15/99): ledmon-libs-1.1.0-3.el9.x86_64.rpm 465 kB/s | 40 kB 00:00 2026-02-26T02:41:59.657 INFO:teuthology.orchestra.run.trial043.stdout:(16/99): cryptsetup-2.8.1-3.el9.x86_64.rpm 1.9 MB/s | 351 kB 00:00 2026-02-26T02:41:59.704 INFO:teuthology.orchestra.run.trial043.stdout:(17/99): libconfig-1.7.2-9.el9.x86_64.rpm 1.2 MB/s | 72 kB 00:00 2026-02-26T02:41:59.725 INFO:teuthology.orchestra.run.trial043.stdout:(18/99): libgfortran-11.5.0-14.el9.x86_64.rpm 11 MB/s | 794 kB 00:00 2026-02-26T02:41:59.756 INFO:teuthology.orchestra.run.trial043.stdout:(19/99): python3-cffi-1.14.5-5.el9.x86_64.rpm 8.1 MB/s | 253 kB 00:00 2026-02-26T02:41:59.766 INFO:teuthology.orchestra.run.trial043.stdout:(20/99): libquadmath-11.5.0-14.el9.x86_64.rpm 2.9 MB/s | 184 kB 00:00 2026-02-26T02:41:59.800 INFO:teuthology.orchestra.run.trial043.stdout:(21/99): python3-ply-3.11-14.el9.noarch.rpm 3.1 MB/s | 106 kB 00:00 2026-02-26T02:41:59.812 INFO:teuthology.orchestra.run.trial043.stdout:(22/99): python3-cryptography-36.0.1-5.el9.x86_ 22 MB/s | 1.2 MB 00:00 2026-02-26T02:41:59.834 INFO:teuthology.orchestra.run.trial043.stdout:(23/99): python3-pycparser-2.20-6.el9.noarch.rp 3.8 MB/s | 135 kB 00:00 2026-02-26T02:41:59.839 INFO:teuthology.orchestra.run.trial043.stdout:(24/99): python3-pyparsing-2.4.7-9.el9.noarch.r 5.4 MB/s | 150 kB 00:00 2026-02-26T02:41:59.867 INFO:teuthology.orchestra.run.trial043.stdout:(25/99): python3-urllib3-1.26.5-7.el9.noarch.rp 7.6 MB/s | 218 kB 00:00 2026-02-26T02:41:59.869 INFO:teuthology.orchestra.run.trial043.stdout:(26/99): python3-requests-2.25.1-10.el9.noarch. 3.6 MB/s | 126 kB 00:00 2026-02-26T02:41:59.894 INFO:teuthology.orchestra.run.trial043.stdout:(27/99): unzip-6.0-59.el9.x86_64.rpm 6.5 MB/s | 182 kB 00:00 2026-02-26T02:41:59.913 INFO:teuthology.orchestra.run.trial043.stdout:(28/99): zip-3.0-35.el9.x86_64.rpm 5.8 MB/s | 266 kB 00:00 2026-02-26T02:41:59.914 INFO:teuthology.orchestra.run.trial043.stdout:(29/99): flexiblas-3.0.4-9.el9.x86_64.rpm 1.5 MB/s | 30 kB 00:00 2026-02-26T02:41:59.918 INFO:teuthology.orchestra.run.trial043.stdout:(30/99): flexiblas-openblas-openmp-3.0.4-9.el9. 3.6 MB/s | 15 kB 00:00 2026-02-26T02:41:59.930 INFO:teuthology.orchestra.run.trial043.stdout:(31/99): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 20 MB/s | 246 kB 00:00 2026-02-26T02:41:59.938 INFO:teuthology.orchestra.run.trial043.stdout:(32/99): lua-5.4.4-4.el9.x86_64.rpm 26 MB/s | 188 kB 00:00 2026-02-26T02:41:59.942 INFO:teuthology.orchestra.run.trial043.stdout:(33/99): openblas-0.3.29-1.el9.x86_64.rpm 9.5 MB/s | 42 kB 00:00 2026-02-26T02:41:59.968 INFO:teuthology.orchestra.run.trial043.stdout:(34/99): openblas-openmp-0.3.29-1.el9.x86_64.rp 209 MB/s | 5.3 MB 00:00 2026-02-26T02:41:59.978 INFO:teuthology.orchestra.run.trial043.stdout:(35/99): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 47 MB/s | 3.0 MB 00:00 2026-02-26T02:41:59.979 INFO:teuthology.orchestra.run.trial043.stdout:(36/99): protobuf-3.14.0-17.el9.x86_64.rpm 85 MB/s | 1.0 MB 00:00 2026-02-26T02:41:59.986 INFO:teuthology.orchestra.run.trial043.stdout:(37/99): python3-devel-3.9.25-3.el9.x86_64.rpm 34 MB/s | 244 kB 00:00 2026-02-26T02:41:59.993 INFO:teuthology.orchestra.run.trial043.stdout:(38/99): python3-jinja2-2.11.3-8.el9.noarch.rpm 35 MB/s | 249 kB 00:00 2026-02-26T02:42:00.000 INFO:teuthology.orchestra.run.trial043.stdout:(39/99): python3-libstoragemgmt-1.10.1-1.el9.x8 27 MB/s | 177 kB 00:00 2026-02-26T02:42:00.010 INFO:teuthology.orchestra.run.trial043.stdout:(40/99): python3-lxml-4.6.5-3.el9.x86_64.rpm 125 MB/s | 1.2 MB 00:00 2026-02-26T02:42:00.017 INFO:teuthology.orchestra.run.trial043.stdout:(41/99): python3-mako-1.1.4-6.el9.noarch.rpm 25 MB/s | 172 kB 00:00 2026-02-26T02:42:00.021 INFO:teuthology.orchestra.run.trial043.stdout:(42/99): python3-markupsafe-1.1.1-12.el9.x86_64 7.5 MB/s | 35 kB 00:00 2026-02-26T02:42:00.038 INFO:teuthology.orchestra.run.trial043.stdout:(43/99): python3-babel-2.9.1-2.el9.noarch.rpm 99 MB/s | 6.0 MB 00:00 2026-02-26T02:42:00.052 INFO:teuthology.orchestra.run.trial043.stdout:(44/99): python3-numpy-1.23.5-2.el9.x86_64.rpm 200 MB/s | 6.1 MB 00:00 2026-02-26T02:42:00.054 INFO:teuthology.orchestra.run.trial043.stdout:(45/99): python3-numpy-f2py-1.23.5-2.el9.x86_64 29 MB/s | 442 kB 00:00 2026-02-26T02:42:00.058 INFO:teuthology.orchestra.run.trial043.stdout:(46/99): python3-packaging-20.9-5.el9.noarch.rp 13 MB/s | 77 kB 00:00 2026-02-26T02:42:00.061 INFO:teuthology.orchestra.run.trial043.stdout:(47/99): python3-protobuf-3.14.0-17.el9.noarch. 36 MB/s | 267 kB 00:00 2026-02-26T02:42:00.064 INFO:teuthology.orchestra.run.trial043.stdout:(48/99): python3-pyasn1-0.4.8-7.el9.noarch.rpm 28 MB/s | 157 kB 00:00 2026-02-26T02:42:00.068 INFO:teuthology.orchestra.run.trial043.stdout:(49/99): python3-pyasn1-modules-0.4.8-7.el9.noa 38 MB/s | 277 kB 00:00 2026-02-26T02:42:00.069 INFO:teuthology.orchestra.run.trial043.stdout:(50/99): python3-requests-oauthlib-1.3.0-12.el9 10 MB/s | 54 kB 00:00 2026-02-26T02:42:00.074 INFO:teuthology.orchestra.run.trial043.stdout:(51/99): python3-toml-0.10.2-6.el9.noarch.rpm 8.5 MB/s | 42 kB 00:00 2026-02-26T02:42:00.079 INFO:teuthology.orchestra.run.trial043.stdout:(52/99): xmlsec1-1.2.29-13.el9.x86_64.rpm 39 MB/s | 189 kB 00:00 2026-02-26T02:42:00.084 INFO:teuthology.orchestra.run.trial043.stdout:(53/99): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 16 MB/s | 90 kB 00:00 2026-02-26T02:42:00.107 INFO:teuthology.orchestra.run.trial043.stdout:(54/99): ceph-osd-19.2.3-687.g8d724284.el9.x86_ 9.9 MB/s | 17 MB 00:01 2026-02-26T02:42:00.133 INFO:teuthology.orchestra.run.trial043.stdout:(55/99): lua-devel-5.4.4-4.el9.x86_64.rpm 454 kB/s | 22 kB 00:00 2026-02-26T02:42:00.180 INFO:teuthology.orchestra.run.trial043.stdout:(56/99): python3-scipy-1.9.3-2.el9.x86_64.rpm 173 MB/s | 19 MB 00:00 2026-02-26T02:42:00.193 INFO:teuthology.orchestra.run.trial043.stdout:(57/99): protobuf-compiler-3.14.0-17.el9.x86_64 9.7 MB/s | 862 kB 00:00 2026-02-26T02:42:00.270 INFO:teuthology.orchestra.run.trial043.stdout:(58/99): grpc-data-1.46.7-10.el9.noarch.rpm 222 kB/s | 19 kB 00:00 2026-02-26T02:42:00.336 INFO:teuthology.orchestra.run.trial043.stdout:(59/99): abseil-cpp-20211102.0-4.el9.x86_64.rpm 2.7 MB/s | 551 kB 00:00 2026-02-26T02:42:00.351 INFO:teuthology.orchestra.run.trial043.stdout:(60/99): luarocks-3.9.2-5.el9.noarch.rpm 958 kB/s | 151 kB 00:00 2026-02-26T02:42:00.358 INFO:teuthology.orchestra.run.trial043.stdout:(61/99): python3-autocommand-2.2.2-8.el9.noarch 1.4 MB/s | 29 kB 00:00 2026-02-26T02:42:00.379 INFO:teuthology.orchestra.run.trial043.stdout:(62/99): python3-asyncssh-2.13.2-5.el9.noarch.r 4.9 MB/s | 548 kB 00:00 2026-02-26T02:42:00.379 INFO:teuthology.orchestra.run.trial043.stdout:(63/99): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 2.0 MB/s | 43 kB 00:00 2026-02-26T02:42:00.383 INFO:teuthology.orchestra.run.trial043.stdout:(64/99): python3-backports-tarfile-1.2.0-1.el9. 1.9 MB/s | 60 kB 00:00 2026-02-26T02:42:00.400 INFO:teuthology.orchestra.run.trial043.stdout:(65/99): python3-cachetools-4.2.4-1.el9.noarch. 1.5 MB/s | 32 kB 00:00 2026-02-26T02:42:00.400 INFO:teuthology.orchestra.run.trial043.stdout:(66/99): python3-certifi-2023.05.07-4.el9.noarc 669 kB/s | 14 kB 00:00 2026-02-26T02:42:00.425 INFO:teuthology.orchestra.run.trial043.stdout:(67/99): python3-cheroot-10.0.1-4.el9.noarch.rp 4.0 MB/s | 173 kB 00:00 2026-02-26T02:42:00.429 INFO:teuthology.orchestra.run.trial043.stdout:(68/99): python3-google-auth-2.45.0-1.el9.noarc 8.7 MB/s | 254 kB 00:00 2026-02-26T02:42:00.430 INFO:teuthology.orchestra.run.trial043.stdout:(69/99): python3-cherrypy-18.6.1-2.el9.noarch.r 12 MB/s | 358 kB 00:00 2026-02-26T02:42:00.458 INFO:teuthology.orchestra.run.trial043.stdout:(70/99): python3-grpcio-tools-1.46.7-10.el9.x86 4.9 MB/s | 144 kB 00:00 2026-02-26T02:42:00.459 INFO:teuthology.orchestra.run.trial043.stdout:(71/99): python3-influxdb-5.3.1-1.el9.noarch.rp 4.7 MB/s | 139 kB 00:00 2026-02-26T02:42:00.482 INFO:teuthology.orchestra.run.trial043.stdout:(72/99): python3-jaraco-8.2.1-3.el9.noarch.rpm 485 kB/s | 11 kB 00:00 2026-02-26T02:42:00.482 INFO:teuthology.orchestra.run.trial043.stdout:(73/99): python3-isodate-0.6.1-3.el9.noarch.rpm 2.3 MB/s | 56 kB 00:00 2026-02-26T02:42:00.503 INFO:teuthology.orchestra.run.trial043.stdout:(74/99): python3-jaraco-classes-3.2.1-5.el9.noa 839 kB/s | 18 kB 00:00 2026-02-26T02:42:00.503 INFO:teuthology.orchestra.run.trial043.stdout:(75/99): python3-jaraco-collections-3.0.0-8.el9 1.1 MB/s | 23 kB 00:00 2026-02-26T02:42:00.524 INFO:teuthology.orchestra.run.trial043.stdout:(76/99): python3-jaraco-context-6.0.1-3.el9.noa 932 kB/s | 20 kB 00:00 2026-02-26T02:42:00.524 INFO:teuthology.orchestra.run.trial043.stdout:(77/99): python3-jaraco-functools-3.5.0-2.el9.n 919 kB/s | 19 kB 00:00 2026-02-26T02:42:00.533 INFO:teuthology.orchestra.run.trial043.stdout:(78/99): python3-grpcio-1.46.7-10.el9.x86_64.rp 19 MB/s | 2.0 MB 00:00 2026-02-26T02:42:00.546 INFO:teuthology.orchestra.run.trial043.stdout:(79/99): python3-jaraco-text-4.0.0-2.el9.noarch 1.2 MB/s | 26 kB 00:00 2026-02-26T02:42:00.558 INFO:teuthology.orchestra.run.trial043.stdout:(80/99): python3-logutils-0.3.5-21.el9.noarch.r 1.9 MB/s | 46 kB 00:00 2026-02-26T02:42:00.568 INFO:teuthology.orchestra.run.trial043.stdout:(81/99): python3-more-itertools-8.12.0-2.el9.no 3.5 MB/s | 79 kB 00:00 2026-02-26T02:42:00.572 INFO:teuthology.orchestra.run.trial043.stdout:(82/99): python3-kubernetes-26.1.0-3.el9.noarch 21 MB/s | 1.0 MB 00:00 2026-02-26T02:42:00.582 INFO:teuthology.orchestra.run.trial043.stdout:(83/99): python3-msgpack-1.0.3-2.el9.x86_64.rpm 3.4 MB/s | 86 kB 00:00 2026-02-26T02:42:00.591 INFO:teuthology.orchestra.run.trial043.stdout:(84/99): python3-natsort-7.1.1-5.el9.noarch.rpm 2.6 MB/s | 58 kB 00:00 2026-02-26T02:42:00.595 INFO:teuthology.orchestra.run.trial043.stdout:(85/99): python3-pecan-1.4.2-3.el9.noarch.rpm 11 MB/s | 272 kB 00:00 2026-02-26T02:42:00.607 INFO:teuthology.orchestra.run.trial043.stdout:(86/99): python3-portend-3.1.0-2.el9.noarch.rpm 683 kB/s | 16 kB 00:00 2026-02-26T02:42:00.613 INFO:teuthology.orchestra.run.trial043.stdout:(87/99): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.9 MB/s | 90 kB 00:00 2026-02-26T02:42:00.616 INFO:teuthology.orchestra.run.trial043.stdout:(88/99): python3-repoze-lru-0.7-16.el9.noarch.r 1.4 MB/s | 31 kB 00:00 2026-02-26T02:42:00.632 INFO:teuthology.orchestra.run.trial043.stdout:(89/99): python3-routes-2.5.1-5.el9.noarch.rpm 7.2 MB/s | 188 kB 00:00 2026-02-26T02:42:00.636 INFO:teuthology.orchestra.run.trial043.stdout:(90/99): python3-rsa-4.9-2.el9.noarch.rpm 2.6 MB/s | 59 kB 00:00 2026-02-26T02:42:00.638 INFO:teuthology.orchestra.run.trial043.stdout:(91/99): python3-saml-1.16.0-1.el9.noarch.rpm 5.7 MB/s | 125 kB 00:00 2026-02-26T02:42:00.657 INFO:teuthology.orchestra.run.trial043.stdout:(92/99): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2026-02-26T02:42:00.658 INFO:teuthology.orchestra.run.trial043.stdout:(93/99): python3-typing-extensions-4.15.0-1.el9 3.8 MB/s | 86 kB 00:00 2026-02-26T02:42:00.661 INFO:teuthology.orchestra.run.trial043.stdout:(94/99): python3-webob-1.8.8-2.el9.noarch.rpm 9.9 MB/s | 230 kB 00:00 2026-02-26T02:42:00.682 INFO:teuthology.orchestra.run.trial043.stdout:(95/99): python3-websocket-client-1.2.3-2.el9.n 3.5 MB/s | 90 kB 00:00 2026-02-26T02:42:00.682 INFO:teuthology.orchestra.run.trial043.stdout:(96/99): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 2.2 MB/s | 48 kB 00:00 2026-02-26T02:42:00.690 INFO:teuthology.orchestra.run.trial043.stdout:(97/99): python3-werkzeug-2.0.3-3.el9.1.noarch. 13 MB/s | 427 kB 00:00 2026-02-26T02:42:00.704 INFO:teuthology.orchestra.run.trial043.stdout:(98/99): python3-zc-lockfile-2.0-10.el9.noarch. 922 kB/s | 20 kB 00:00 2026-02-26T02:42:00.706 INFO:teuthology.orchestra.run.trial043.stdout:(99/99): python3-xmltodict-0.12.0-15.el9.noarch 891 kB/s | 22 kB 00:00 2026-02-26T02:42:00.709 INFO:teuthology.orchestra.run.trial043.stdout:-------------------------------------------------------------------------------- 2026-02-26T02:42:00.709 INFO:teuthology.orchestra.run.trial043.stdout:Total 27 MB/s | 95 MB 00:03 2026-02-26T02:42:01.135 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction check 2026-02-26T02:42:01.172 INFO:teuthology.orchestra.run.trial043.stdout:Transaction check succeeded. 2026-02-26T02:42:01.173 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction test 2026-02-26T02:42:01.595 INFO:teuthology.orchestra.run.trial043.stdout:Transaction test succeeded. 2026-02-26T02:42:01.595 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction 2026-02-26T02:42:02.052 INFO:teuthology.orchestra.run.trial043.stdout: Preparing : 1/1 2026-02-26T02:42:02.053 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/99 2026-02-26T02:42:02.085 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/99 2026-02-26T02:42:02.089 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 3/99 2026-02-26T02:42:02.111 INFO:teuthology.orchestra.run.trial043.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 4/99 2026-02-26T02:42:02.117 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 5/99 2026-02-26T02:42:02.150 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 6/99 2026-02-26T02:42:02.159 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 7/99 2026-02-26T02:42:02.164 INFO:teuthology.orchestra.run.trial043.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 8/99 2026-02-26T02:42:02.167 INFO:teuthology.orchestra.run.trial043.stdout: Installing : lua-5.4.4-4.el9.x86_64 9/99 2026-02-26T02:42:02.172 INFO:teuthology.orchestra.run.trial043.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 10/99 2026-02-26T02:42:02.196 INFO:teuthology.orchestra.run.trial043.stdout: Installing : unzip-6.0-59.el9.x86_64 11/99 2026-02-26T02:42:02.208 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 12/99 2026-02-26T02:42:02.212 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 13/99 2026-02-26T02:42:02.218 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 14/99 2026-02-26T02:42:02.220 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 15/99 2026-02-26T02:42:02.231 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 16/99 2026-02-26T02:42:02.237 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 17/99 2026-02-26T02:42:02.261 INFO:teuthology.orchestra.run.trial043.stdout: Installing : zip-3.0-35.el9.x86_64 18/99 2026-02-26T02:42:02.264 INFO:teuthology.orchestra.run.trial043.stdout: Installing : luarocks-3.9.2-5.el9.noarch 19/99 2026-02-26T02:42:02.271 INFO:teuthology.orchestra.run.trial043.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 20/99 2026-02-26T02:42:02.296 INFO:teuthology.orchestra.run.trial043.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 21/99 2026-02-26T02:42:02.347 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 22/99 2026-02-26T02:42:02.360 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 23/99 2026-02-26T02:42:02.362 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-rsa-4.9-2.el9.noarch 24/99 2026-02-26T02:42:02.367 INFO:teuthology.orchestra.run.trial043.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 25/99 2026-02-26T02:42:02.372 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 26/99 2026-02-26T02:42:02.377 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 27/99 2026-02-26T02:42:02.380 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 28/99 2026-02-26T02:42:02.395 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 29/99 2026-02-26T02:42:02.416 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 30/99 2026-02-26T02:42:02.420 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 31/99 2026-02-26T02:42:02.425 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 32/99 2026-02-26T02:42:02.437 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 33/99 2026-02-26T02:42:02.446 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 34/99 2026-02-26T02:42:02.452 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 35/99 2026-02-26T02:42:02.472 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 36/99 2026-02-26T02:42:02.481 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 37/99 2026-02-26T02:42:02.536 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 38/99 2026-02-26T02:42:02.548 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 39/99 2026-02-26T02:42:02.563 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 40/99 2026-02-26T02:42:02.568 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 41/99 2026-02-26T02:42:02.576 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 42/99 2026-02-26T02:42:02.616 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 43/99 2026-02-26T02:42:02.942 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 44/99 2026-02-26T02:42:02.952 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 45/99 2026-02-26T02:42:02.956 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 46/99 2026-02-26T02:42:02.963 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 47/99 2026-02-26T02:42:02.965 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 48/99 2026-02-26T02:42:02.993 INFO:teuthology.orchestra.run.trial043.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 49/99 2026-02-26T02:42:03.037 INFO:teuthology.orchestra.run.trial043.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 50/99 2026-02-26T02:42:03.043 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 51/99 2026-02-26T02:42:03.047 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 52/99 2026-02-26T02:42:03.052 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 53/99 2026-02-26T02:42:03.056 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 54/99 2026-02-26T02:42:03.063 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 55/99 2026-02-26T02:42:03.067 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 56/99 2026-02-26T02:42:03.095 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 57/99 2026-02-26T02:42:03.105 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 58/99 2026-02-26T02:42:03.140 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 59/99 2026-02-26T02:42:03.365 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 60/99 2026-02-26T02:42:03.389 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 61/99 2026-02-26T02:42:03.393 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 62/99 2026-02-26T02:42:03.451 INFO:teuthology.orchestra.run.trial043.stdout: Installing : openblas-0.3.29-1.el9.x86_64 63/99 2026-02-26T02:42:03.453 INFO:teuthology.orchestra.run.trial043.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 64/99 2026-02-26T02:42:03.474 INFO:teuthology.orchestra.run.trial043.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 65/99 2026-02-26T02:42:03.802 INFO:teuthology.orchestra.run.trial043.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 66/99 2026-02-26T02:42:03.875 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 67/99 2026-02-26T02:42:04.551 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 68/99 2026-02-26T02:42:04.568 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 69/99 2026-02-26T02:42:04.579 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 70/99 2026-02-26T02:42:04.585 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-packaging-20.9-5.el9.noarch 71/99 2026-02-26T02:42:04.599 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-ply-3.11-14.el9.noarch 72/99 2026-02-26T02:42:04.615 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 73/99 2026-02-26T02:42:04.690 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 74/99 2026-02-26T02:42:04.699 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 75/99 2026-02-26T02:42:04.723 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 76/99 2026-02-26T02:42:04.753 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 77/99 2026-02-26T02:42:04.804 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 78/99 2026-02-26T02:42:04.809 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 79/99 2026-02-26T02:42:04.813 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 80/99 2026-02-26T02:42:04.814 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 81/99 2026-02-26T02:42:04.832 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 82/99 2026-02-26T02:42:04.832 INFO:teuthology.orchestra.run.trial043.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-26T02:42:04.832 INFO:teuthology.orchestra.run.trial043.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-26T02:42:04.832 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:04.839 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 82/99 2026-02-26T02:42:04.864 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 82/99 2026-02-26T02:42:04.865 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-26T02:42:04.865 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:04.899 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 83/99 2026-02-26T02:42:04.963 INFO:teuthology.orchestra.run.trial043.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 84/99 2026-02-26T02:42:04.965 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-volume-2:19.2.3-687.g8d724284.el9.noarch 85/99 2026-02-26T02:42:04.979 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-volume-2:19.2.3-687.g8d724284.el9.noarch 85/99 2026-02-26T02:42:04.979 INFO:teuthology.orchestra.run.trial043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T02:42:04.980 INFO:teuthology.orchestra.run.trial043.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-26T02:42:04.980 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:05.678 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-osd-2:19.2.3-687.g8d724284.el9.x86_64 86/99 2026-02-26T02:42:05.703 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-osd-2:19.2.3-687.g8d724284.el9.x86_64 86/99 2026-02-26T02:42:05.703 INFO:teuthology.orchestra.run.trial043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T02:42:05.703 INFO:teuthology.orchestra.run.trial043.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-26T02:42:05.703 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-26T02:42:05.703 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-26T02:42:05.703 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:05.759 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: cephadm-2:19.2.3-687.g8d724284.el9.noarch 87/99 2026-02-26T02:42:05.761 INFO:teuthology.orchestra.run.trial043.stdout: Installing : cephadm-2:19.2.3-687.g8d724284.el9.noarch 87/99 2026-02-26T02:42:05.766 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-prometheus-alerts-2:19.2.3-687.g8d724284.el9. 88/99 2026-02-26T02:42:05.786 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-grafana-dashboards-2:19.2.3-687.g8d724284.el9 89/99 2026-02-26T02:42:05.788 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-mgr-cephadm-2:19.2.3-687.g8d724284.el9.noarch 90/99 2026-02-26T02:42:06.282 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.2.3-687.g8d724284.el9.noarch 90/99 2026-02-26T02:42:06.285 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-mgr-dashboard-2:19.2.3-687.g8d724284.el9.noar 91/99 2026-02-26T02:42:06.740 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.2.3-687.g8d724284.el9.noar 91/99 2026-02-26T02:42:06.741 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-mgr-diskprediction-local-2:19.2.3-687.g8d7242 92/99 2026-02-26T02:42:06.753 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.2.3-687.g8d7242 92/99 2026-02-26T02:42:06.754 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-mgr-k8sevents-2:19.2.3-687.g8d724284.el9.noar 93/99 2026-02-26T02:42:06.810 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-mgr-k8sevents-2:19.2.3-687.g8d724284.el9.noar 93/99 2026-02-26T02:42:06.860 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-mgr-modules-core-2:19.2.3-687.g8d724284.el9.n 94/99 2026-02-26T02:42:06.861 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-mgr-2:19.2.3-687.g8d724284.el9.x86_64 95/99 2026-02-26T02:42:06.882 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-mgr-2:19.2.3-687.g8d724284.el9.x86_64 95/99 2026-02-26T02:42:06.882 INFO:teuthology.orchestra.run.trial043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T02:42:06.882 INFO:teuthology.orchestra.run.trial043.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-26T02:42:06.882 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-26T02:42:06.882 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-26T02:42:06.882 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:06.894 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-mgr-rook-2:19.2.3-687.g8d724284.el9.noarch 96/99 2026-02-26T02:42:07.108 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-mgr-rook-2:19.2.3-687.g8d724284.el9.noarch 96/99 2026-02-26T02:42:07.109 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-mon-2:19.2.3-687.g8d724284.el9.x86_64 97/99 2026-02-26T02:42:07.130 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-mon-2:19.2.3-687.g8d724284.el9.x86_64 97/99 2026-02-26T02:42:07.130 INFO:teuthology.orchestra.run.trial043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T02:42:07.130 INFO:teuthology.orchestra.run.trial043.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-26T02:42:07.130 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-26T02:42:07.131 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-26T02:42:07.131 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:07.231 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-mds-2:19.2.3-687.g8d724284.el9.x86_64 98/99 2026-02-26T02:42:07.253 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-mds-2:19.2.3-687.g8d724284.el9.x86_64 98/99 2026-02-26T02:42:07.253 INFO:teuthology.orchestra.run.trial043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T02:42:07.253 INFO:teuthology.orchestra.run.trial043.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-26T02:42:07.253 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-26T02:42:07.253 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-26T02:42:07.253 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:07.272 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-2:19.2.3-687.g8d724284.el9.x86_64 99/99 2026-02-26T02:42:07.958 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-2:19.2.3-687.g8d724284.el9.x86_64 99/99 2026-02-26T02:42:07.958 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-2:19.2.3-687.g8d724284.el9.x86_64 1/99 2026-02-26T02:42:07.958 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-mds-2:19.2.3-687.g8d724284.el9.x86_64 2/99 2026-02-26T02:42:07.958 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-mgr-2:19.2.3-687.g8d724284.el9.x86_64 3/99 2026-02-26T02:42:07.958 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-mon-2:19.2.3-687.g8d724284.el9.x86_64 4/99 2026-02-26T02:42:07.958 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-osd-2:19.2.3-687.g8d724284.el9.x86_64 5/99 2026-02-26T02:42:07.958 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-grafana-dashboards-2:19.2.3-687.g8d724284.el9 6/99 2026-02-26T02:42:07.958 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-mgr-cephadm-2:19.2.3-687.g8d724284.el9.noarch 7/99 2026-02-26T02:42:07.958 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-mgr-dashboard-2:19.2.3-687.g8d724284.el9.noar 8/99 2026-02-26T02:42:07.958 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.2.3-687.g8d7242 9/99 2026-02-26T02:42:07.958 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-mgr-k8sevents-2:19.2.3-687.g8d724284.el9.noar 10/99 2026-02-26T02:42:07.958 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-mgr-modules-core-2:19.2.3-687.g8d724284.el9.n 11/99 2026-02-26T02:42:07.958 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-mgr-rook-2:19.2.3-687.g8d724284.el9.noarch 12/99 2026-02-26T02:42:07.958 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-prometheus-alerts-2:19.2.3-687.g8d724284.el9. 13/99 2026-02-26T02:42:07.959 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-volume-2:19.2.3-687.g8d724284.el9.noarch 14/99 2026-02-26T02:42:07.959 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : cephadm-2:19.2.3-687.g8d724284.el9.noarch 15/99 2026-02-26T02:42:07.959 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 16/99 2026-02-26T02:42:07.959 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 17/99 2026-02-26T02:42:07.959 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 18/99 2026-02-26T02:42:07.959 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 19/99 2026-02-26T02:42:07.959 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 20/99 2026-02-26T02:42:07.959 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 21/99 2026-02-26T02:42:07.959 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 22/99 2026-02-26T02:42:07.959 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-ply-3.11-14.el9.noarch 23/99 2026-02-26T02:42:07.959 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 24/99 2026-02-26T02:42:07.959 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 25/99 2026-02-26T02:42:07.959 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 26/99 2026-02-26T02:42:07.959 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 27/99 2026-02-26T02:42:07.959 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : unzip-6.0-59.el9.x86_64 28/99 2026-02-26T02:42:07.959 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : zip-3.0-35.el9.x86_64 29/99 2026-02-26T02:42:07.959 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 30/99 2026-02-26T02:42:07.959 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 31/99 2026-02-26T02:42:07.959 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 32/99 2026-02-26T02:42:07.959 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 33/99 2026-02-26T02:42:07.959 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : lua-5.4.4-4.el9.x86_64 34/99 2026-02-26T02:42:07.959 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 35/99 2026-02-26T02:42:07.959 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 36/99 2026-02-26T02:42:07.959 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 37/99 2026-02-26T02:42:07.962 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 38/99 2026-02-26T02:42:07.962 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 39/99 2026-02-26T02:42:07.962 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 40/99 2026-02-26T02:42:07.962 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 41/99 2026-02-26T02:42:07.962 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 42/99 2026-02-26T02:42:07.962 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 43/99 2026-02-26T02:42:07.962 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 44/99 2026-02-26T02:42:07.962 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 45/99 2026-02-26T02:42:07.962 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 46/99 2026-02-26T02:42:07.962 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 47/99 2026-02-26T02:42:07.962 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 48/99 2026-02-26T02:42:07.962 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 49/99 2026-02-26T02:42:07.962 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 50/99 2026-02-26T02:42:07.962 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 51/99 2026-02-26T02:42:07.962 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 52/99 2026-02-26T02:42:07.962 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 53/99 2026-02-26T02:42:07.962 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 54/99 2026-02-26T02:42:07.962 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 55/99 2026-02-26T02:42:07.962 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 56/99 2026-02-26T02:42:07.962 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 57/99 2026-02-26T02:42:07.962 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 58/99 2026-02-26T02:42:07.962 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 59/99 2026-02-26T02:42:07.962 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 60/99 2026-02-26T02:42:07.962 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 61/99 2026-02-26T02:42:07.962 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 62/99 2026-02-26T02:42:07.962 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 63/99 2026-02-26T02:42:07.962 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 64/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 65/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 66/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 67/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 68/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 69/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 70/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 71/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 72/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 73/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 74/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 75/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 76/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 77/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 78/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 79/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 80/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 81/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 85/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 86/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 88/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 89/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 90/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 91/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 92/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 93/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 94/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 95/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 96/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 97/99 2026-02-26T02:42:07.963 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 98/99 2026-02-26T02:42:08.031 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 99/99 2026-02-26T02:42:08.031 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:08.031 INFO:teuthology.orchestra.run.trial043.stdout:Installed: 2026-02-26T02:42:08.031 INFO:teuthology.orchestra.run.trial043.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-26T02:42:08.031 INFO:teuthology.orchestra.run.trial043.stdout: ceph-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T02:42:08.031 INFO:teuthology.orchestra.run.trial043.stdout: ceph-grafana-dashboards-2:19.2.3-687.g8d724284.el9.noarch 2026-02-26T02:42:08.031 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mds-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mgr-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mgr-cephadm-2:19.2.3-687.g8d724284.el9.noarch 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mgr-dashboard-2:19.2.3-687.g8d724284.el9.noarch 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mgr-diskprediction-local-2:19.2.3-687.g8d724284.el9.noarch 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mgr-k8sevents-2:19.2.3-687.g8d724284.el9.noarch 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mgr-modules-core-2:19.2.3-687.g8d724284.el9.noarch 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mgr-rook-2:19.2.3-687.g8d724284.el9.noarch 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mon-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: ceph-osd-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: ceph-prometheus-alerts-2:19.2.3-687.g8d724284.el9.noarch 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: ceph-volume-2:19.2.3-687.g8d724284.el9.noarch 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: cephadm-2:19.2.3-687.g8d724284.el9.noarch 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: lua-5.4.4-4.el9.x86_64 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-26T02:42:08.032 INFO:teuthology.orchestra.run.trial043.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-logutils-0.3.5-21.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-mako-1.1.4-6.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-pecan-1.4.2-3.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-ply-3.11-14.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-webob-1.8.8-2.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-26T02:42:08.033 INFO:teuthology.orchestra.run.trial043.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2026-02-26T02:42:08.034 INFO:teuthology.orchestra.run.trial043.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-26T02:42:08.034 INFO:teuthology.orchestra.run.trial043.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-26T02:42:08.034 INFO:teuthology.orchestra.run.trial043.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-26T02:42:08.034 INFO:teuthology.orchestra.run.trial043.stdout: unzip-6.0-59.el9.x86_64 2026-02-26T02:42:08.034 INFO:teuthology.orchestra.run.trial043.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-26T02:42:08.034 INFO:teuthology.orchestra.run.trial043.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-26T02:42:08.034 INFO:teuthology.orchestra.run.trial043.stdout: zip-3.0-35.el9.x86_64 2026-02-26T02:42:08.034 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:08.034 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-26T02:42:08.086 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-base 2026-02-26T02:42:08.406 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:34 ago on Thu 26 Feb 2026 02:41:34 AM UTC. 2026-02-26T02:42:08.472 INFO:teuthology.orchestra.run.trial043.stdout:Package ceph-base-2:19.2.3-687.g8d724284.el9.x86_64 is already installed. 2026-02-26T02:42:08.491 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-26T02:42:08.491 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-26T02:42:08.491 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-26T02:42:08.513 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install cephadm 2026-02-26T02:42:08.825 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:34 ago on Thu 26 Feb 2026 02:41:34 AM UTC. 2026-02-26T02:42:08.891 INFO:teuthology.orchestra.run.trial043.stdout:Package cephadm-2:19.2.3-687.g8d724284.el9.noarch is already installed. 2026-02-26T02:42:08.910 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-26T02:42:08.910 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-26T02:42:08.910 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-26T02:42:08.932 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-immutable-object-cache 2026-02-26T02:42:09.249 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:35 ago on Thu 26 Feb 2026 02:41:34 AM UTC. 2026-02-26T02:42:09.334 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-26T02:42:09.334 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-26T02:42:09.334 INFO:teuthology.orchestra.run.trial043.stdout: Package Arch Version Repo Size 2026-02-26T02:42:09.334 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-26T02:42:09.335 INFO:teuthology.orchestra.run.trial043.stdout:Installing: 2026-02-26T02:42:09.335 INFO:teuthology.orchestra.run.trial043.stdout: ceph-immutable-object-cache x86_64 2:19.2.3-687.g8d724284.el9 ceph 145 k 2026-02-26T02:42:09.335 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:09.335 INFO:teuthology.orchestra.run.trial043.stdout:Transaction Summary 2026-02-26T02:42:09.335 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-26T02:42:09.335 INFO:teuthology.orchestra.run.trial043.stdout:Install 1 Package 2026-02-26T02:42:09.335 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:09.335 INFO:teuthology.orchestra.run.trial043.stdout:Total download size: 145 k 2026-02-26T02:42:09.335 INFO:teuthology.orchestra.run.trial043.stdout:Installed size: 431 k 2026-02-26T02:42:09.335 INFO:teuthology.orchestra.run.trial043.stdout:Downloading Packages: 2026-02-26T02:42:09.674 INFO:teuthology.orchestra.run.trial043.stdout:ceph-immutable-object-cache-19.2.3-687.g8d72428 429 kB/s | 145 kB 00:00 2026-02-26T02:42:09.674 INFO:teuthology.orchestra.run.trial043.stdout:-------------------------------------------------------------------------------- 2026-02-26T02:42:09.674 INFO:teuthology.orchestra.run.trial043.stdout:Total 428 kB/s | 145 kB 00:00 2026-02-26T02:42:09.674 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction check 2026-02-26T02:42:09.678 INFO:teuthology.orchestra.run.trial043.stdout:Transaction check succeeded. 2026-02-26T02:42:09.678 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction test 2026-02-26T02:42:09.710 INFO:teuthology.orchestra.run.trial043.stdout:Transaction test succeeded. 2026-02-26T02:42:09.711 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction 2026-02-26T02:42:09.766 INFO:teuthology.orchestra.run.trial043.stdout: Preparing : 1/1 2026-02-26T02:42:09.767 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-immutable-object-cache-2:19.2.3-687.g8d724284.e 1/1 2026-02-26T02:42:09.790 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.2.3-687.g8d724284.e 1/1 2026-02-26T02:42:09.790 INFO:teuthology.orchestra.run.trial043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T02:42:09.790 INFO:teuthology.orchestra.run.trial043.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-26T02:42:09.790 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:10.108 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-immutable-object-cache-2:19.2.3-687.g8d724284.e 1/1 2026-02-26T02:42:10.108 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:10.108 INFO:teuthology.orchestra.run.trial043.stdout:Installed: 2026-02-26T02:42:10.108 INFO:teuthology.orchestra.run.trial043.stdout: ceph-immutable-object-cache-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T02:42:10.108 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:10.108 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-26T02:42:10.135 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-mgr 2026-02-26T02:42:10.454 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:36 ago on Thu 26 Feb 2026 02:41:34 AM UTC. 2026-02-26T02:42:10.520 INFO:teuthology.orchestra.run.trial043.stdout:Package ceph-mgr-2:19.2.3-687.g8d724284.el9.x86_64 is already installed. 2026-02-26T02:42:10.539 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-26T02:42:10.540 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-26T02:42:10.540 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-26T02:42:10.562 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-mgr-dashboard 2026-02-26T02:42:10.882 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:36 ago on Thu 26 Feb 2026 02:41:34 AM UTC. 2026-02-26T02:42:10.947 INFO:teuthology.orchestra.run.trial043.stdout:Package ceph-mgr-dashboard-2:19.2.3-687.g8d724284.el9.noarch is already installed. 2026-02-26T02:42:10.966 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-26T02:42:10.966 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-26T02:42:10.967 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-26T02:42:10.988 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-26T02:42:11.306 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:37 ago on Thu 26 Feb 2026 02:41:34 AM UTC. 2026-02-26T02:42:11.374 INFO:teuthology.orchestra.run.trial043.stdout:Package ceph-mgr-diskprediction-local-2:19.2.3-687.g8d724284.el9.noarch is already installed. 2026-02-26T02:42:11.392 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-26T02:42:11.393 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-26T02:42:11.393 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-26T02:42:11.415 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-mgr-rook 2026-02-26T02:42:11.735 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:37 ago on Thu 26 Feb 2026 02:41:34 AM UTC. 2026-02-26T02:42:11.800 INFO:teuthology.orchestra.run.trial043.stdout:Package ceph-mgr-rook-2:19.2.3-687.g8d724284.el9.noarch is already installed. 2026-02-26T02:42:11.819 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-26T02:42:11.819 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-26T02:42:11.819 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-26T02:42:11.840 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-mgr-cephadm 2026-02-26T02:42:12.157 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:38 ago on Thu 26 Feb 2026 02:41:34 AM UTC. 2026-02-26T02:42:12.222 INFO:teuthology.orchestra.run.trial043.stdout:Package ceph-mgr-cephadm-2:19.2.3-687.g8d724284.el9.noarch is already installed. 2026-02-26T02:42:12.241 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-26T02:42:12.242 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-26T02:42:12.242 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-26T02:42:12.263 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-fuse 2026-02-26T02:42:12.577 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:38 ago on Thu 26 Feb 2026 02:41:34 AM UTC. 2026-02-26T02:42:12.664 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-26T02:42:12.664 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-26T02:42:12.664 INFO:teuthology.orchestra.run.trial043.stdout: Package Architecture Version Repository Size 2026-02-26T02:42:12.664 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-26T02:42:12.664 INFO:teuthology.orchestra.run.trial043.stdout:Installing: 2026-02-26T02:42:12.664 INFO:teuthology.orchestra.run.trial043.stdout: ceph-fuse x86_64 2:19.2.3-687.g8d724284.el9 ceph 1.2 M 2026-02-26T02:42:12.664 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:12.664 INFO:teuthology.orchestra.run.trial043.stdout:Transaction Summary 2026-02-26T02:42:12.665 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-26T02:42:12.665 INFO:teuthology.orchestra.run.trial043.stdout:Install 1 Package 2026-02-26T02:42:12.665 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:12.665 INFO:teuthology.orchestra.run.trial043.stdout:Total download size: 1.2 M 2026-02-26T02:42:12.665 INFO:teuthology.orchestra.run.trial043.stdout:Installed size: 3.6 M 2026-02-26T02:42:12.665 INFO:teuthology.orchestra.run.trial043.stdout:Downloading Packages: 2026-02-26T02:42:12.980 INFO:teuthology.orchestra.run.trial043.stdout:ceph-fuse-19.2.3-687.g8d724284.el9.x86_64.rpm 3.6 MB/s | 1.2 MB 00:00 2026-02-26T02:42:12.981 INFO:teuthology.orchestra.run.trial043.stdout:-------------------------------------------------------------------------------- 2026-02-26T02:42:12.981 INFO:teuthology.orchestra.run.trial043.stdout:Total 3.6 MB/s | 1.2 MB 00:00 2026-02-26T02:42:12.981 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction check 2026-02-26T02:42:12.984 INFO:teuthology.orchestra.run.trial043.stdout:Transaction check succeeded. 2026-02-26T02:42:12.985 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction test 2026-02-26T02:42:13.018 INFO:teuthology.orchestra.run.trial043.stdout:Transaction test succeeded. 2026-02-26T02:42:13.019 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction 2026-02-26T02:42:13.120 INFO:teuthology.orchestra.run.trial043.stdout: Preparing : 1/1 2026-02-26T02:42:13.138 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-fuse-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T02:42:13.365 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-fuse-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T02:42:13.426 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-fuse-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T02:42:13.426 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:13.426 INFO:teuthology.orchestra.run.trial043.stdout:Installed: 2026-02-26T02:42:13.427 INFO:teuthology.orchestra.run.trial043.stdout: ceph-fuse-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T02:42:13.427 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:13.427 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-26T02:42:13.454 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-volume 2026-02-26T02:42:13.772 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:39 ago on Thu 26 Feb 2026 02:41:34 AM UTC. 2026-02-26T02:42:13.838 INFO:teuthology.orchestra.run.trial043.stdout:Package ceph-volume-2:19.2.3-687.g8d724284.el9.noarch is already installed. 2026-02-26T02:42:13.857 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-26T02:42:13.857 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-26T02:42:13.857 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-26T02:42:13.879 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install librados-devel 2026-02-26T02:42:14.197 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:40 ago on Thu 26 Feb 2026 02:41:34 AM UTC. 2026-02-26T02:42:14.281 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-26T02:42:14.281 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-26T02:42:14.282 INFO:teuthology.orchestra.run.trial043.stdout: Package Arch Version Repo Size 2026-02-26T02:42:14.282 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-26T02:42:14.282 INFO:teuthology.orchestra.run.trial043.stdout:Installing: 2026-02-26T02:42:14.282 INFO:teuthology.orchestra.run.trial043.stdout: librados-devel x86_64 2:19.2.3-687.g8d724284.el9 ceph 126 k 2026-02-26T02:42:14.282 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:14.282 INFO:teuthology.orchestra.run.trial043.stdout:Transaction Summary 2026-02-26T02:42:14.282 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-26T02:42:14.282 INFO:teuthology.orchestra.run.trial043.stdout:Install 1 Package 2026-02-26T02:42:14.282 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:14.282 INFO:teuthology.orchestra.run.trial043.stdout:Total download size: 126 k 2026-02-26T02:42:14.282 INFO:teuthology.orchestra.run.trial043.stdout:Installed size: 456 k 2026-02-26T02:42:14.283 INFO:teuthology.orchestra.run.trial043.stdout:Downloading Packages: 2026-02-26T02:42:14.509 INFO:teuthology.orchestra.run.trial043.stdout:librados-devel-19.2.3-687.g8d724284.el9.x86_64. 555 kB/s | 126 kB 00:00 2026-02-26T02:42:14.510 INFO:teuthology.orchestra.run.trial043.stdout:-------------------------------------------------------------------------------- 2026-02-26T02:42:14.510 INFO:teuthology.orchestra.run.trial043.stdout:Total 552 kB/s | 126 kB 00:00 2026-02-26T02:42:14.511 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction check 2026-02-26T02:42:14.513 INFO:teuthology.orchestra.run.trial043.stdout:Transaction check succeeded. 2026-02-26T02:42:14.513 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction test 2026-02-26T02:42:14.545 INFO:teuthology.orchestra.run.trial043.stdout:Transaction test succeeded. 2026-02-26T02:42:14.546 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction 2026-02-26T02:42:14.602 INFO:teuthology.orchestra.run.trial043.stdout: Preparing : 1/1 2026-02-26T02:42:14.620 INFO:teuthology.orchestra.run.trial043.stdout: Installing : librados-devel-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T02:42:14.706 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: librados-devel-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T02:42:14.769 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : librados-devel-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T02:42:14.770 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:14.770 INFO:teuthology.orchestra.run.trial043.stdout:Installed: 2026-02-26T02:42:14.770 INFO:teuthology.orchestra.run.trial043.stdout: librados-devel-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T02:42:14.770 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:14.770 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-26T02:42:14.798 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install libcephfs2 2026-02-26T02:42:15.117 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:41 ago on Thu 26 Feb 2026 02:41:34 AM UTC. 2026-02-26T02:42:15.184 INFO:teuthology.orchestra.run.trial043.stdout:Package libcephfs2-2:19.2.3-687.g8d724284.el9.x86_64 is already installed. 2026-02-26T02:42:15.203 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-26T02:42:15.204 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-26T02:42:15.204 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-26T02:42:15.225 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install libcephfs-devel 2026-02-26T02:42:15.539 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:41 ago on Thu 26 Feb 2026 02:41:34 AM UTC. 2026-02-26T02:42:15.626 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-26T02:42:15.626 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-26T02:42:15.626 INFO:teuthology.orchestra.run.trial043.stdout: Package Arch Version Repo Size 2026-02-26T02:42:15.626 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-26T02:42:15.627 INFO:teuthology.orchestra.run.trial043.stdout:Installing: 2026-02-26T02:42:15.627 INFO:teuthology.orchestra.run.trial043.stdout: libcephfs-devel x86_64 2:19.2.3-687.g8d724284.el9 ceph 34 k 2026-02-26T02:42:15.627 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:15.627 INFO:teuthology.orchestra.run.trial043.stdout:Transaction Summary 2026-02-26T02:42:15.627 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-26T02:42:15.627 INFO:teuthology.orchestra.run.trial043.stdout:Install 1 Package 2026-02-26T02:42:15.627 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:15.627 INFO:teuthology.orchestra.run.trial043.stdout:Total download size: 34 k 2026-02-26T02:42:15.627 INFO:teuthology.orchestra.run.trial043.stdout:Installed size: 153 k 2026-02-26T02:42:15.628 INFO:teuthology.orchestra.run.trial043.stdout:Downloading Packages: 2026-02-26T02:42:15.800 INFO:teuthology.orchestra.run.trial043.stdout:libcephfs-devel-19.2.3-687.g8d724284.el9.x86_64 193 kB/s | 34 kB 00:00 2026-02-26T02:42:15.801 INFO:teuthology.orchestra.run.trial043.stdout:-------------------------------------------------------------------------------- 2026-02-26T02:42:15.801 INFO:teuthology.orchestra.run.trial043.stdout:Total 192 kB/s | 34 kB 00:00 2026-02-26T02:42:15.801 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction check 2026-02-26T02:42:15.803 INFO:teuthology.orchestra.run.trial043.stdout:Transaction check succeeded. 2026-02-26T02:42:15.803 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction test 2026-02-26T02:42:15.805 INFO:teuthology.orchestra.run.trial043.stdout:Transaction test succeeded. 2026-02-26T02:42:15.806 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction 2026-02-26T02:42:15.826 INFO:teuthology.orchestra.run.trial043.stdout: Preparing : 1/1 2026-02-26T02:42:15.843 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libcephfs-devel-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T02:42:15.903 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: libcephfs-devel-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T02:42:15.964 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libcephfs-devel-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T02:42:15.965 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:15.965 INFO:teuthology.orchestra.run.trial043.stdout:Installed: 2026-02-26T02:42:15.965 INFO:teuthology.orchestra.run.trial043.stdout: libcephfs-devel-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T02:42:15.965 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:15.965 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-26T02:42:15.993 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install librados2 2026-02-26T02:42:16.308 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:42 ago on Thu 26 Feb 2026 02:41:34 AM UTC. 2026-02-26T02:42:16.374 INFO:teuthology.orchestra.run.trial043.stdout:Package librados2-2:19.2.3-687.g8d724284.el9.x86_64 is already installed. 2026-02-26T02:42:16.393 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-26T02:42:16.393 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-26T02:42:16.394 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-26T02:42:16.415 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install librbd1 2026-02-26T02:42:16.731 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:42 ago on Thu 26 Feb 2026 02:41:34 AM UTC. 2026-02-26T02:42:16.797 INFO:teuthology.orchestra.run.trial043.stdout:Package librbd1-2:19.2.3-687.g8d724284.el9.x86_64 is already installed. 2026-02-26T02:42:16.816 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-26T02:42:16.817 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-26T02:42:16.817 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-26T02:42:16.838 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install python3-rados 2026-02-26T02:42:17.160 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:43 ago on Thu 26 Feb 2026 02:41:34 AM UTC. 2026-02-26T02:42:17.226 INFO:teuthology.orchestra.run.trial043.stdout:Package python3-rados-2:19.2.3-687.g8d724284.el9.x86_64 is already installed. 2026-02-26T02:42:17.245 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-26T02:42:17.246 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-26T02:42:17.246 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-26T02:42:17.268 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install python3-rgw 2026-02-26T02:42:17.587 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:43 ago on Thu 26 Feb 2026 02:41:34 AM UTC. 2026-02-26T02:42:17.654 INFO:teuthology.orchestra.run.trial043.stdout:Package python3-rgw-2:19.2.3-687.g8d724284.el9.x86_64 is already installed. 2026-02-26T02:42:17.673 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-26T02:42:17.673 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-26T02:42:17.673 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-26T02:42:17.696 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install python3-cephfs 2026-02-26T02:42:18.012 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:44 ago on Thu 26 Feb 2026 02:41:34 AM UTC. 2026-02-26T02:42:18.077 INFO:teuthology.orchestra.run.trial043.stdout:Package python3-cephfs-2:19.2.3-687.g8d724284.el9.x86_64 is already installed. 2026-02-26T02:42:18.096 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-26T02:42:18.096 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-26T02:42:18.096 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-26T02:42:18.117 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install python3-rbd 2026-02-26T02:42:18.430 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:44 ago on Thu 26 Feb 2026 02:41:34 AM UTC. 2026-02-26T02:42:18.496 INFO:teuthology.orchestra.run.trial043.stdout:Package python3-rbd-2:19.2.3-687.g8d724284.el9.x86_64 is already installed. 2026-02-26T02:42:18.515 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-26T02:42:18.516 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-26T02:42:18.516 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-26T02:42:18.537 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install rbd-fuse 2026-02-26T02:42:18.852 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:44 ago on Thu 26 Feb 2026 02:41:34 AM UTC. 2026-02-26T02:42:18.939 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-26T02:42:18.939 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-26T02:42:18.939 INFO:teuthology.orchestra.run.trial043.stdout: Package Architecture Version Repository Size 2026-02-26T02:42:18.939 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-26T02:42:18.939 INFO:teuthology.orchestra.run.trial043.stdout:Installing: 2026-02-26T02:42:18.939 INFO:teuthology.orchestra.run.trial043.stdout: rbd-fuse x86_64 2:19.2.3-687.g8d724284.el9 ceph 85 k 2026-02-26T02:42:18.939 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:18.939 INFO:teuthology.orchestra.run.trial043.stdout:Transaction Summary 2026-02-26T02:42:18.939 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-26T02:42:18.940 INFO:teuthology.orchestra.run.trial043.stdout:Install 1 Package 2026-02-26T02:42:18.940 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:18.940 INFO:teuthology.orchestra.run.trial043.stdout:Total download size: 85 k 2026-02-26T02:42:18.940 INFO:teuthology.orchestra.run.trial043.stdout:Installed size: 227 k 2026-02-26T02:42:18.940 INFO:teuthology.orchestra.run.trial043.stdout:Downloading Packages: 2026-02-26T02:42:19.067 INFO:teuthology.orchestra.run.trial043.stdout:rbd-fuse-19.2.3-687.g8d724284.el9.x86_64.rpm 666 kB/s | 85 kB 00:00 2026-02-26T02:42:19.067 INFO:teuthology.orchestra.run.trial043.stdout:-------------------------------------------------------------------------------- 2026-02-26T02:42:19.067 INFO:teuthology.orchestra.run.trial043.stdout:Total 662 kB/s | 85 kB 00:00 2026-02-26T02:42:19.068 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction check 2026-02-26T02:42:19.070 INFO:teuthology.orchestra.run.trial043.stdout:Transaction check succeeded. 2026-02-26T02:42:19.070 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction test 2026-02-26T02:42:19.101 INFO:teuthology.orchestra.run.trial043.stdout:Transaction test succeeded. 2026-02-26T02:42:19.102 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction 2026-02-26T02:42:19.153 INFO:teuthology.orchestra.run.trial043.stdout: Preparing : 1/1 2026-02-26T02:42:19.171 INFO:teuthology.orchestra.run.trial043.stdout: Installing : rbd-fuse-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T02:42:19.254 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: rbd-fuse-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T02:42:19.317 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : rbd-fuse-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T02:42:19.317 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:19.317 INFO:teuthology.orchestra.run.trial043.stdout:Installed: 2026-02-26T02:42:19.317 INFO:teuthology.orchestra.run.trial043.stdout: rbd-fuse-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T02:42:19.317 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:19.317 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-26T02:42:19.345 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install rbd-mirror 2026-02-26T02:42:19.659 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:45 ago on Thu 26 Feb 2026 02:41:34 AM UTC. 2026-02-26T02:42:19.744 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-26T02:42:19.744 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-26T02:42:19.744 INFO:teuthology.orchestra.run.trial043.stdout: Package Arch Version Repo Size 2026-02-26T02:42:19.745 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-26T02:42:19.745 INFO:teuthology.orchestra.run.trial043.stdout:Installing: 2026-02-26T02:42:19.745 INFO:teuthology.orchestra.run.trial043.stdout: rbd-mirror x86_64 2:19.2.3-687.g8d724284.el9 ceph 3.1 M 2026-02-26T02:42:19.745 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:19.745 INFO:teuthology.orchestra.run.trial043.stdout:Transaction Summary 2026-02-26T02:42:19.745 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-26T02:42:19.745 INFO:teuthology.orchestra.run.trial043.stdout:Install 1 Package 2026-02-26T02:42:19.745 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:19.745 INFO:teuthology.orchestra.run.trial043.stdout:Total download size: 3.1 M 2026-02-26T02:42:19.745 INFO:teuthology.orchestra.run.trial043.stdout:Installed size: 13 M 2026-02-26T02:42:19.745 INFO:teuthology.orchestra.run.trial043.stdout:Downloading Packages: 2026-02-26T02:42:20.100 INFO:teuthology.orchestra.run.trial043.stdout:rbd-mirror-19.2.3-687.g8d724284.el9.x86_64.rpm 8.8 MB/s | 3.1 MB 00:00 2026-02-26T02:42:20.101 INFO:teuthology.orchestra.run.trial043.stdout:-------------------------------------------------------------------------------- 2026-02-26T02:42:20.101 INFO:teuthology.orchestra.run.trial043.stdout:Total 8.7 MB/s | 3.1 MB 00:00 2026-02-26T02:42:20.101 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction check 2026-02-26T02:42:20.105 INFO:teuthology.orchestra.run.trial043.stdout:Transaction check succeeded. 2026-02-26T02:42:20.105 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction test 2026-02-26T02:42:20.144 INFO:teuthology.orchestra.run.trial043.stdout:Transaction test succeeded. 2026-02-26T02:42:20.145 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction 2026-02-26T02:42:20.338 INFO:teuthology.orchestra.run.trial043.stdout: Preparing : 1/1 2026-02-26T02:42:20.339 INFO:teuthology.orchestra.run.trial043.stdout: Installing : rbd-mirror-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T02:42:20.361 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: rbd-mirror-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T02:42:20.361 INFO:teuthology.orchestra.run.trial043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T02:42:20.361 INFO:teuthology.orchestra.run.trial043.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-26T02:42:20.361 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-26T02:42:20.361 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-26T02:42:20.362 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:20.672 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : rbd-mirror-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T02:42:20.672 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:20.672 INFO:teuthology.orchestra.run.trial043.stdout:Installed: 2026-02-26T02:42:20.672 INFO:teuthology.orchestra.run.trial043.stdout: rbd-mirror-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T02:42:20.672 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:20.672 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-26T02:42:20.701 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install rbd-nbd 2026-02-26T02:42:21.021 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:47 ago on Thu 26 Feb 2026 02:41:34 AM UTC. 2026-02-26T02:42:21.107 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-26T02:42:21.107 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-26T02:42:21.107 INFO:teuthology.orchestra.run.trial043.stdout: Package Architecture Version Repository Size 2026-02-26T02:42:21.107 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-26T02:42:21.107 INFO:teuthology.orchestra.run.trial043.stdout:Installing: 2026-02-26T02:42:21.107 INFO:teuthology.orchestra.run.trial043.stdout: rbd-nbd x86_64 2:19.2.3-687.g8d724284.el9 ceph 170 k 2026-02-26T02:42:21.107 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:21.108 INFO:teuthology.orchestra.run.trial043.stdout:Transaction Summary 2026-02-26T02:42:21.108 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-26T02:42:21.108 INFO:teuthology.orchestra.run.trial043.stdout:Install 1 Package 2026-02-26T02:42:21.108 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:21.108 INFO:teuthology.orchestra.run.trial043.stdout:Total download size: 170 k 2026-02-26T02:42:21.108 INFO:teuthology.orchestra.run.trial043.stdout:Installed size: 490 k 2026-02-26T02:42:21.108 INFO:teuthology.orchestra.run.trial043.stdout:Downloading Packages: 2026-02-26T02:42:21.273 INFO:teuthology.orchestra.run.trial043.stdout:rbd-nbd-19.2.3-687.g8d724284.el9.x86_64.rpm 1.0 MB/s | 170 kB 00:00 2026-02-26T02:42:21.273 INFO:teuthology.orchestra.run.trial043.stdout:-------------------------------------------------------------------------------- 2026-02-26T02:42:21.273 INFO:teuthology.orchestra.run.trial043.stdout:Total 1.0 MB/s | 170 kB 00:00 2026-02-26T02:42:21.273 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction check 2026-02-26T02:42:21.277 INFO:teuthology.orchestra.run.trial043.stdout:Transaction check succeeded. 2026-02-26T02:42:21.277 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction test 2026-02-26T02:42:21.308 INFO:teuthology.orchestra.run.trial043.stdout:Transaction test succeeded. 2026-02-26T02:42:21.308 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction 2026-02-26T02:42:21.364 INFO:teuthology.orchestra.run.trial043.stdout: Preparing : 1/1 2026-02-26T02:42:21.382 INFO:teuthology.orchestra.run.trial043.stdout: Installing : rbd-nbd-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T02:42:21.465 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: rbd-nbd-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T02:42:21.527 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : rbd-nbd-2:19.2.3-687.g8d724284.el9.x86_64 1/1 2026-02-26T02:42:21.527 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:21.527 INFO:teuthology.orchestra.run.trial043.stdout:Installed: 2026-02-26T02:42:21.527 INFO:teuthology.orchestra.run.trial043.stdout: rbd-nbd-2:19.2.3-687.g8d724284.el9.x86_64 2026-02-26T02:42:21.527 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:21.528 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-26T02:42:21.560 DEBUG:teuthology.parallel:result is None 2026-02-26T02:42:21.560 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5 2026-02-26T02:42:21.692 DEBUG:teuthology.orchestra.run.trial043:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-26T02:42:21.712 INFO:teuthology.orchestra.run.trial043.stdout:19.2.3-687.g8d724284.el9 2026-02-26T02:42:21.712 INFO:teuthology.packaging:The installed version of ceph is 19.2.3-687.g8d724284.el9 2026-02-26T02:42:21.712 INFO:teuthology.task.install:The correct ceph version 19.2.3-687.g8d724284 is installed. 2026-02-26T02:42:21.714 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-26T02:42:21.714 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-26T02:42:21.714 DEBUG:teuthology.orchestra.run.trial043:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-26T02:42:21.776 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-26T02:42:21.777 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-26T02:42:21.777 DEBUG:teuthology.orchestra.run.trial043:> sudo dd of=/usr/bin/daemon-helper 2026-02-26T02:42:21.840 DEBUG:teuthology.orchestra.run.trial043:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-26T02:42:21.904 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-26T02:42:21.904 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-26T02:42:21.905 DEBUG:teuthology.orchestra.run.trial043:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-26T02:42:21.968 DEBUG:teuthology.orchestra.run.trial043:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-26T02:42:22.031 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-26T02:42:22.032 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-26T02:42:22.032 DEBUG:teuthology.orchestra.run.trial043:> sudo dd of=/usr/bin/stdin-killer 2026-02-26T02:42:22.095 DEBUG:teuthology.orchestra.run.trial043:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-26T02:42:22.158 INFO:teuthology.run_tasks:Running task ceph... 2026-02-26T02:42:22.238 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-26T02:42:22.238 DEBUG:teuthology.orchestra.run.trial043:> sudo chmod 777 /var/log/ceph 2026-02-26T02:42:22.263 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-26T02:42:22.263 DEBUG:teuthology.orchestra.run.trial043:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-26T02:42:22.327 INFO:tasks.ceph:Creating extra log directories... 2026-02-26T02:42:22.327 DEBUG:teuthology.orchestra.run.trial043:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-26T02:42:22.393 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-26T02:42:22.393 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 3, 'mon mgr beacon grace': 90, 'ms bind msgr2': False, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['Reduced data availability', 'OSD_.*DOWN', '\\(RECENT_CRASH\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_DEGRADED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2026-02-26T02:42:22.393 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-26_01:51:16-rados-wip-yuri9-testing-2026-02-25-1600-squid-distro-default-trial/71042', 'branch': 'wip-yuri9-testing-2026-02-25-1600-squid', 'description': 'rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'flavor': 'default', 'job_id': '71042', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'yuriw-2026-02-26_01:51:16-rados-wip-yuri9-testing-2026-02-25-1600-squid-distro-default-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri9-testing-2026-02-25-1600-squid'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 3, 'mon mgr beacon grace': 90, 'ms bind msgr2': False, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'mon_bind_msgr2': False, 'sha1': '8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri9-testing-2026-02-25-1600-squid', 'sha1': '8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5'}}, 'owner': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'priority': 69, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['client.0', 'mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2']], 'seed': 1525, 'sha1': '8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri9-testing-2026-02-25-1600-squid', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5', 'targets': {'trial043.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPri+9SLxA35ppeGNRG0esqvoxCp406uXTc4O1OSKl3xWTZxdp+Ayg0GMvcRXGSupYz8guymaqfPqIjHwCoN/Gc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'log-ignorelist': ['Reduced data availability', 'OSD_.*DOWN', '\\(RECENT_CRASH\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_DEGRADED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'conf': {'global': {'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 3, 'mon mgr beacon grace': 90, 'ms bind msgr2': False, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'mon_bind_msgr2': False, 'sha1': '8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['rados/test_crash.sh']}}}, {'ceph.restart': ['osd.*']}, {'exec': {'mon.a': ['find $TESTDIR/archive/coredump -type f -exec rm -f {} \\;']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6e3d0419ccca0e75a03e08feb708946e198236f4', 'timestamp': '2026-02-26_01:51:16', 'tube': 'trial', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699'} 2026-02-26T02:42:22.393 DEBUG:teuthology.orchestra.run.trial043:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-26T02:42:22.445 DEBUG:teuthology.orchestra.run.trial043:> sudo install -d -m0777 -- /var/run/ceph 2026-02-26T02:42:22.509 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-26T02:42:22.509 DEBUG:teuthology.orchestra.run.trial043:> dd if=/scratch_devs of=/dev/stdout 2026-02-26T02:42:22.564 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-26T02:42:22.564 DEBUG:teuthology.orchestra.run.trial043:> stat /dev/vg_nvme/lv_1 2026-02-26T02:42:22.622 INFO:teuthology.orchestra.run.trial043.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-26T02:42:22.622 INFO:teuthology.orchestra.run.trial043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T02:42:22.623 INFO:teuthology.orchestra.run.trial043.stdout:Device: 6h/6d Inode: 1311 Links: 1 2026-02-26T02:42:22.623 INFO:teuthology.orchestra.run.trial043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T02:42:22.623 INFO:teuthology.orchestra.run.trial043.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T02:42:22.623 INFO:teuthology.orchestra.run.trial043.stdout:Access: 2026-02-26 02:42:20.509636763 +0000 2026-02-26T02:42:22.623 INFO:teuthology.orchestra.run.trial043.stdout:Modify: 2026-02-26 02:41:41.885068599 +0000 2026-02-26T02:42:22.623 INFO:teuthology.orchestra.run.trial043.stdout:Change: 2026-02-26 02:41:41.885068599 +0000 2026-02-26T02:42:22.623 INFO:teuthology.orchestra.run.trial043.stdout: Birth: 2026-02-26 02:41:41.885068599 +0000 2026-02-26T02:42:22.624 DEBUG:teuthology.orchestra.run.trial043:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-26T02:42:22.685 INFO:teuthology.orchestra.run.trial043.stderr:1+0 records in 2026-02-26T02:42:22.685 INFO:teuthology.orchestra.run.trial043.stderr:1+0 records out 2026-02-26T02:42:22.685 INFO:teuthology.orchestra.run.trial043.stderr:512 bytes copied, 0.000163972 s, 3.1 MB/s 2026-02-26T02:42:22.687 DEBUG:teuthology.orchestra.run.trial043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-26T02:42:22.744 DEBUG:teuthology.orchestra.run.trial043:> stat /dev/vg_nvme/lv_2 2026-02-26T02:42:22.800 INFO:teuthology.orchestra.run.trial043.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-26T02:42:22.800 INFO:teuthology.orchestra.run.trial043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T02:42:22.800 INFO:teuthology.orchestra.run.trial043.stdout:Device: 6h/6d Inode: 1306 Links: 1 2026-02-26T02:42:22.800 INFO:teuthology.orchestra.run.trial043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T02:42:22.800 INFO:teuthology.orchestra.run.trial043.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T02:42:22.800 INFO:teuthology.orchestra.run.trial043.stdout:Access: 2026-02-26 02:42:20.509636763 +0000 2026-02-26T02:42:22.800 INFO:teuthology.orchestra.run.trial043.stdout:Modify: 2026-02-26 02:41:41.885068599 +0000 2026-02-26T02:42:22.800 INFO:teuthology.orchestra.run.trial043.stdout:Change: 2026-02-26 02:41:41.885068599 +0000 2026-02-26T02:42:22.800 INFO:teuthology.orchestra.run.trial043.stdout: Birth: 2026-02-26 02:41:41.885068599 +0000 2026-02-26T02:42:22.800 DEBUG:teuthology.orchestra.run.trial043:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-26T02:42:22.863 INFO:teuthology.orchestra.run.trial043.stderr:1+0 records in 2026-02-26T02:42:22.863 INFO:teuthology.orchestra.run.trial043.stderr:1+0 records out 2026-02-26T02:42:22.863 INFO:teuthology.orchestra.run.trial043.stderr:512 bytes copied, 0.000135259 s, 3.8 MB/s 2026-02-26T02:42:22.864 DEBUG:teuthology.orchestra.run.trial043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-26T02:42:22.920 DEBUG:teuthology.orchestra.run.trial043:> stat /dev/vg_nvme/lv_3 2026-02-26T02:42:22.978 INFO:teuthology.orchestra.run.trial043.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-26T02:42:22.979 INFO:teuthology.orchestra.run.trial043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T02:42:22.979 INFO:teuthology.orchestra.run.trial043.stdout:Device: 6h/6d Inode: 1300 Links: 1 2026-02-26T02:42:22.979 INFO:teuthology.orchestra.run.trial043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T02:42:22.979 INFO:teuthology.orchestra.run.trial043.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T02:42:22.979 INFO:teuthology.orchestra.run.trial043.stdout:Access: 2026-02-26 02:42:20.509636763 +0000 2026-02-26T02:42:22.979 INFO:teuthology.orchestra.run.trial043.stdout:Modify: 2026-02-26 02:41:41.884068637 +0000 2026-02-26T02:42:22.979 INFO:teuthology.orchestra.run.trial043.stdout:Change: 2026-02-26 02:41:41.884068637 +0000 2026-02-26T02:42:22.979 INFO:teuthology.orchestra.run.trial043.stdout: Birth: 2026-02-26 02:41:41.884068637 +0000 2026-02-26T02:42:22.980 DEBUG:teuthology.orchestra.run.trial043:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-26T02:42:23.041 INFO:teuthology.orchestra.run.trial043.stderr:1+0 records in 2026-02-26T02:42:23.041 INFO:teuthology.orchestra.run.trial043.stderr:1+0 records out 2026-02-26T02:42:23.042 INFO:teuthology.orchestra.run.trial043.stderr:512 bytes copied, 0.000176124 s, 2.9 MB/s 2026-02-26T02:42:23.043 DEBUG:teuthology.orchestra.run.trial043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-26T02:42:23.100 DEBUG:teuthology.orchestra.run.trial043:> stat /dev/vg_nvme/lv_4 2026-02-26T02:42:23.156 INFO:teuthology.orchestra.run.trial043.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-26T02:42:23.157 INFO:teuthology.orchestra.run.trial043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T02:42:23.157 INFO:teuthology.orchestra.run.trial043.stdout:Device: 6h/6d Inode: 1292 Links: 1 2026-02-26T02:42:23.157 INFO:teuthology.orchestra.run.trial043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T02:42:23.157 INFO:teuthology.orchestra.run.trial043.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T02:42:23.157 INFO:teuthology.orchestra.run.trial043.stdout:Access: 2026-02-26 02:42:20.510636726 +0000 2026-02-26T02:42:23.157 INFO:teuthology.orchestra.run.trial043.stdout:Modify: 2026-02-26 02:41:41.884068637 +0000 2026-02-26T02:42:23.157 INFO:teuthology.orchestra.run.trial043.stdout:Change: 2026-02-26 02:41:41.884068637 +0000 2026-02-26T02:42:23.157 INFO:teuthology.orchestra.run.trial043.stdout: Birth: 2026-02-26 02:41:41.884068637 +0000 2026-02-26T02:42:23.158 DEBUG:teuthology.orchestra.run.trial043:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-26T02:42:23.219 INFO:teuthology.orchestra.run.trial043.stderr:1+0 records in 2026-02-26T02:42:23.220 INFO:teuthology.orchestra.run.trial043.stderr:1+0 records out 2026-02-26T02:42:23.220 INFO:teuthology.orchestra.run.trial043.stderr:512 bytes copied, 0.00014546 s, 3.5 MB/s 2026-02-26T02:42:23.221 DEBUG:teuthology.orchestra.run.trial043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-26T02:42:23.278 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T02:42:23.278 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial043.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-26T02:42:23.278 INFO:tasks.ceph:Generating config... 2026-02-26T02:42:23.281 INFO:tasks.ceph:[global] mon client directed command retry = 5 2026-02-26T02:42:23.281 INFO:tasks.ceph:[global] mon client hunt interval max multiple = 2 2026-02-26T02:42:23.281 INFO:tasks.ceph:[global] mon election default strategy = 3 2026-02-26T02:42:23.282 INFO:tasks.ceph:[global] mon mgr beacon grace = 90 2026-02-26T02:42:23.282 INFO:tasks.ceph:[global] ms bind msgr2 = False 2026-02-26T02:42:23.282 INFO:tasks.ceph:[global] ms inject socket failures = 1000 2026-02-26T02:42:23.282 INFO:tasks.ceph:[global] ms type = async 2026-02-26T02:42:23.282 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-26T02:42:23.282 INFO:tasks.ceph:[mgr] debug monc = 10 2026-02-26T02:42:23.282 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-26T02:42:23.282 INFO:tasks.ceph:[mon] debug mon = 20 2026-02-26T02:42:23.282 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-26T02:42:23.282 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-26T02:42:23.283 INFO:tasks.ceph:[mon] mon scrub interval = 300 2026-02-26T02:42:23.283 INFO:tasks.ceph:[osd] bdev async discard = True 2026-02-26T02:42:23.283 INFO:tasks.ceph:[osd] bdev enable discard = True 2026-02-26T02:42:23.283 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2026-02-26T02:42:23.283 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2026-02-26T02:42:23.283 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2026-02-26T02:42:23.283 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2026-02-26T02:42:23.283 INFO:tasks.ceph:[osd] debug bluefs = 20 2026-02-26T02:42:23.283 INFO:tasks.ceph:[osd] debug bluestore = 20 2026-02-26T02:42:23.283 INFO:tasks.ceph:[osd] debug ms = 1 2026-02-26T02:42:23.283 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-26T02:42:23.283 INFO:tasks.ceph:[osd] debug rocksdb = 10 2026-02-26T02:42:23.284 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2026-02-26T02:42:23.284 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2026-02-26T02:42:23.284 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2026-02-26T02:42:23.284 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2026-02-26T02:42:23.284 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2026-02-26T02:42:23.284 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2026-02-26T02:42:23.284 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2026-02-26T02:42:23.284 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2026-02-26T02:42:23.284 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2026-02-26T02:42:23.284 INFO:tasks.ceph:[osd] osd op queue = debug_random 2026-02-26T02:42:23.284 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2026-02-26T02:42:23.285 INFO:tasks.ceph:Setting up mon.a... 2026-02-26T02:42:23.285 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-26T02:42:23.352 INFO:teuthology.orchestra.run.trial043.stdout:creating /etc/ceph/ceph.keyring 2026-02-26T02:42:23.355 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-26T02:42:23.431 DEBUG:teuthology.orchestra.run.trial043:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-26T02:42:23.494 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:10.20.193.43:6789]')] 2026-02-26T02:42:23.494 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:10.20.193.43:6789]', 'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 3, 'mon mgr beacon grace': 90, 'ms bind msgr2': False, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-26T02:42:23.494 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-26T02:42:23.495 DEBUG:teuthology.orchestra.run.trial043:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-26T02:42:23.551 DEBUG:teuthology.orchestra.run.trial043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:10.20.193.43:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2026-02-26T02:42:23.622 INFO:teuthology.orchestra.run.trial043.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-26T02:42:23.622 INFO:teuthology.orchestra.run.trial043.stdout:monmaptool: generated fsid 6dea32ae-afd8-4e41-a06e-2ec4e4797235 2026-02-26T02:42:23.623 INFO:teuthology.orchestra.run.trial043.stdout:setting min_mon_release = quincy 2026-02-26T02:42:23.623 INFO:teuthology.orchestra.run.trial043.stdout:epoch 0 2026-02-26T02:42:23.623 INFO:teuthology.orchestra.run.trial043.stdout:fsid 6dea32ae-afd8-4e41-a06e-2ec4e4797235 2026-02-26T02:42:23.623 INFO:teuthology.orchestra.run.trial043.stdout:last_changed 2026-02-26T02:42:23.620592+0000 2026-02-26T02:42:23.623 INFO:teuthology.orchestra.run.trial043.stdout:created 2026-02-26T02:42:23.620592+0000 2026-02-26T02:42:23.623 INFO:teuthology.orchestra.run.trial043.stdout:min_mon_release 17 (quincy) 2026-02-26T02:42:23.623 INFO:teuthology.orchestra.run.trial043.stdout:election_strategy: 3 2026-02-26T02:42:23.623 INFO:teuthology.orchestra.run.trial043.stdout:0: v1:10.20.193.43:6789/0 mon.a 2026-02-26T02:42:23.623 INFO:teuthology.orchestra.run.trial043.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-26T02:42:23.625 DEBUG:teuthology.orchestra.run.trial043:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-26T02:42:23.681 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 6dea32ae-afd8-4e41-a06e-2ec4e4797235... 2026-02-26T02:42:23.682 DEBUG:teuthology.orchestra.run.trial043:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-26T02:42:23.761 INFO:teuthology.orchestra.run.trial043.stdout:[global] 2026-02-26T02:42:23.761 INFO:teuthology.orchestra.run.trial043.stdout: chdir = "" 2026-02-26T02:42:23.761 INFO:teuthology.orchestra.run.trial043.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-26T02:42:23.762 INFO:teuthology.orchestra.run.trial043.stdout: auth supported = cephx 2026-02-26T02:42:23.762 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:23.762 INFO:teuthology.orchestra.run.trial043.stdout: filestore xattr use omap = true 2026-02-26T02:42:23.762 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:23.762 INFO:teuthology.orchestra.run.trial043.stdout: mon clock drift allowed = 1.000 2026-02-26T02:42:23.762 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:23.762 INFO:teuthology.orchestra.run.trial043.stdout: osd crush chooseleaf type = 0 2026-02-26T02:42:23.762 INFO:teuthology.orchestra.run.trial043.stdout: auth debug = true 2026-02-26T02:42:23.762 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:23.762 INFO:teuthology.orchestra.run.trial043.stdout: ms die on old message = true 2026-02-26T02:42:23.763 INFO:teuthology.orchestra.run.trial043.stdout: ms die on bug = true 2026-02-26T02:42:23.763 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:23.763 INFO:teuthology.orchestra.run.trial043.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-26T02:42:23.763 INFO:teuthology.orchestra.run.trial043.stdout: mon pg warn max object skew = 0 2026-02-26T02:42:23.763 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:23.763 INFO:teuthology.orchestra.run.trial043.stdout: # disable pg_autoscaler by default for new pools 2026-02-26T02:42:23.763 INFO:teuthology.orchestra.run.trial043.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-26T02:42:23.763 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:23.763 INFO:teuthology.orchestra.run.trial043.stdout: osd pool default size = 2 2026-02-26T02:42:23.764 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:23.764 INFO:teuthology.orchestra.run.trial043.stdout: mon osd allow primary affinity = true 2026-02-26T02:42:23.764 INFO:teuthology.orchestra.run.trial043.stdout: mon osd allow pg remap = true 2026-02-26T02:42:23.764 INFO:teuthology.orchestra.run.trial043.stdout: mon warn on legacy crush tunables = false 2026-02-26T02:42:23.764 INFO:teuthology.orchestra.run.trial043.stdout: mon warn on crush straw calc version zero = false 2026-02-26T02:42:23.764 INFO:teuthology.orchestra.run.trial043.stdout: mon warn on no sortbitwise = false 2026-02-26T02:42:23.764 INFO:teuthology.orchestra.run.trial043.stdout: mon warn on osd down out interval zero = false 2026-02-26T02:42:23.764 INFO:teuthology.orchestra.run.trial043.stdout: mon warn on too few osds = false 2026-02-26T02:42:23.764 INFO:teuthology.orchestra.run.trial043.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-26T02:42:23.764 INFO:teuthology.orchestra.run.trial043.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-26T02:42:23.764 INFO:teuthology.orchestra.run.trial043.stdout: mon_allow_pool_size_one = true 2026-02-26T02:42:23.765 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:23.765 INFO:teuthology.orchestra.run.trial043.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-26T02:42:23.765 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:23.765 INFO:teuthology.orchestra.run.trial043.stdout: osd default data pool replay window = 5 2026-02-26T02:42:23.765 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:23.765 INFO:teuthology.orchestra.run.trial043.stdout: mon allow pool delete = true 2026-02-26T02:42:23.765 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:23.765 INFO:teuthology.orchestra.run.trial043.stdout: mon cluster log file level = debug 2026-02-26T02:42:23.765 INFO:teuthology.orchestra.run.trial043.stdout: debug asserts on shutdown = true 2026-02-26T02:42:23.765 INFO:teuthology.orchestra.run.trial043.stdout: mon health detail to clog = false 2026-02-26T02:42:23.765 INFO:teuthology.orchestra.run.trial043.stdout: mon host = [v1:10.20.193.43:6789] 2026-02-26T02:42:23.765 INFO:teuthology.orchestra.run.trial043.stdout: mon client directed command retry = 5 2026-02-26T02:42:23.766 INFO:teuthology.orchestra.run.trial043.stdout: mon client hunt interval max multiple = 2 2026-02-26T02:42:23.766 INFO:teuthology.orchestra.run.trial043.stdout: mon election default strategy = 3 2026-02-26T02:42:23.766 INFO:teuthology.orchestra.run.trial043.stdout: mon mgr beacon grace = 90 2026-02-26T02:42:23.766 INFO:teuthology.orchestra.run.trial043.stdout: ms bind msgr2 = False 2026-02-26T02:42:23.766 INFO:teuthology.orchestra.run.trial043.stdout: ms inject socket failures = 1000 2026-02-26T02:42:23.766 INFO:teuthology.orchestra.run.trial043.stdout: ms type = async 2026-02-26T02:42:23.766 INFO:teuthology.orchestra.run.trial043.stdout: fsid = 6dea32ae-afd8-4e41-a06e-2ec4e4797235 2026-02-26T02:42:23.766 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:23.766 INFO:teuthology.orchestra.run.trial043.stdout:[osd] 2026-02-26T02:42:23.766 INFO:teuthology.orchestra.run.trial043.stdout: osd journal size = 100 2026-02-26T02:42:23.766 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:23.766 INFO:teuthology.orchestra.run.trial043.stdout: osd scrub load threshold = 5.0 2026-02-26T02:42:23.766 INFO:teuthology.orchestra.run.trial043.stdout: osd scrub max interval = 600 2026-02-26T02:42:23.767 INFO:teuthology.orchestra.run.trial043.stdout: osd mclock profile = high_recovery_ops 2026-02-26T02:42:23.767 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:23.767 INFO:teuthology.orchestra.run.trial043.stdout: osd recover clone overlap = true 2026-02-26T02:42:23.767 INFO:teuthology.orchestra.run.trial043.stdout: osd recovery max chunk = 1048576 2026-02-26T02:42:23.767 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:23.767 INFO:teuthology.orchestra.run.trial043.stdout: osd debug shutdown = true 2026-02-26T02:42:23.767 INFO:teuthology.orchestra.run.trial043.stdout: osd debug op order = true 2026-02-26T02:42:23.767 INFO:teuthology.orchestra.run.trial043.stdout: osd debug verify stray on activate = true 2026-02-26T02:42:23.767 INFO:teuthology.orchestra.run.trial043.stdout: osd debug trim objects = true 2026-02-26T02:42:23.767 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:23.767 INFO:teuthology.orchestra.run.trial043.stdout: osd open classes on start = true 2026-02-26T02:42:23.768 INFO:teuthology.orchestra.run.trial043.stdout: osd debug pg log writeout = true 2026-02-26T02:42:23.768 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:23.768 INFO:teuthology.orchestra.run.trial043.stdout: osd deep scrub update digest min age = 30 2026-02-26T02:42:23.768 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:23.768 INFO:teuthology.orchestra.run.trial043.stdout: osd map max advance = 10 2026-02-26T02:42:23.768 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:23.768 INFO:teuthology.orchestra.run.trial043.stdout: journal zero on create = true 2026-02-26T02:42:23.768 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:23.768 INFO:teuthology.orchestra.run.trial043.stdout: filestore ondisk finisher threads = 3 2026-02-26T02:42:23.768 INFO:teuthology.orchestra.run.trial043.stdout: filestore apply finisher threads = 3 2026-02-26T02:42:23.768 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:23.768 INFO:teuthology.orchestra.run.trial043.stdout: bdev debug aio = true 2026-02-26T02:42:23.768 INFO:teuthology.orchestra.run.trial043.stdout: osd debug misdirected ops = true 2026-02-26T02:42:23.769 INFO:teuthology.orchestra.run.trial043.stdout: bdev async discard = True 2026-02-26T02:42:23.769 INFO:teuthology.orchestra.run.trial043.stdout: bdev enable discard = True 2026-02-26T02:42:23.769 INFO:teuthology.orchestra.run.trial043.stdout: bluestore allocator = bitmap 2026-02-26T02:42:23.769 INFO:teuthology.orchestra.run.trial043.stdout: bluestore block size = 96636764160 2026-02-26T02:42:23.769 INFO:teuthology.orchestra.run.trial043.stdout: bluestore fsck on mount = True 2026-02-26T02:42:23.769 INFO:teuthology.orchestra.run.trial043.stdout: bluestore zero block detection = True 2026-02-26T02:42:23.769 INFO:teuthology.orchestra.run.trial043.stdout: debug bluefs = 20 2026-02-26T02:42:23.769 INFO:teuthology.orchestra.run.trial043.stdout: debug bluestore = 20 2026-02-26T02:42:23.769 INFO:teuthology.orchestra.run.trial043.stdout: debug ms = 1 2026-02-26T02:42:23.769 INFO:teuthology.orchestra.run.trial043.stdout: debug osd = 20 2026-02-26T02:42:23.769 INFO:teuthology.orchestra.run.trial043.stdout: debug rocksdb = 10 2026-02-26T02:42:23.769 INFO:teuthology.orchestra.run.trial043.stdout: mon osd backfillfull_ratio = 0.85 2026-02-26T02:42:23.769 INFO:teuthology.orchestra.run.trial043.stdout: mon osd full ratio = 0.9 2026-02-26T02:42:23.769 INFO:teuthology.orchestra.run.trial043.stdout: mon osd nearfull ratio = 0.8 2026-02-26T02:42:23.770 INFO:teuthology.orchestra.run.trial043.stdout: osd debug verify cached snaps = True 2026-02-26T02:42:23.770 INFO:teuthology.orchestra.run.trial043.stdout: osd debug verify missing on start = True 2026-02-26T02:42:23.770 INFO:teuthology.orchestra.run.trial043.stdout: osd failsafe full ratio = 0.95 2026-02-26T02:42:23.770 INFO:teuthology.orchestra.run.trial043.stdout: osd mclock override recovery settings = True 2026-02-26T02:42:23.770 INFO:teuthology.orchestra.run.trial043.stdout: osd objectstore = bluestore 2026-02-26T02:42:23.770 INFO:teuthology.orchestra.run.trial043.stdout: osd op queue = debug_random 2026-02-26T02:42:23.770 INFO:teuthology.orchestra.run.trial043.stdout: osd op queue cut off = debug_random 2026-02-26T02:42:23.770 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:23.770 INFO:teuthology.orchestra.run.trial043.stdout:[mgr] 2026-02-26T02:42:23.770 INFO:teuthology.orchestra.run.trial043.stdout: debug ms = 1 2026-02-26T02:42:23.770 INFO:teuthology.orchestra.run.trial043.stdout: debug mgr = 20 2026-02-26T02:42:23.770 INFO:teuthology.orchestra.run.trial043.stdout: debug mon = 20 2026-02-26T02:42:23.770 INFO:teuthology.orchestra.run.trial043.stdout: debug auth = 20 2026-02-26T02:42:23.771 INFO:teuthology.orchestra.run.trial043.stdout: mon reweight min pgs per osd = 4 2026-02-26T02:42:23.771 INFO:teuthology.orchestra.run.trial043.stdout: mon reweight min bytes per osd = 10 2026-02-26T02:42:23.771 INFO:teuthology.orchestra.run.trial043.stdout: mgr/telemetry/nag = false 2026-02-26T02:42:23.771 INFO:teuthology.orchestra.run.trial043.stdout: debug monc = 10 2026-02-26T02:42:23.771 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:23.771 INFO:teuthology.orchestra.run.trial043.stdout:[mon] 2026-02-26T02:42:23.771 INFO:teuthology.orchestra.run.trial043.stdout: debug ms = 1 2026-02-26T02:42:23.771 INFO:teuthology.orchestra.run.trial043.stdout: debug mon = 20 2026-02-26T02:42:23.771 INFO:teuthology.orchestra.run.trial043.stdout: debug paxos = 20 2026-02-26T02:42:23.771 INFO:teuthology.orchestra.run.trial043.stdout: debug auth = 20 2026-02-26T02:42:23.771 INFO:teuthology.orchestra.run.trial043.stdout: mon data avail warn = 5 2026-02-26T02:42:23.771 INFO:teuthology.orchestra.run.trial043.stdout: mon mgr mkfs grace = 240 2026-02-26T02:42:23.772 INFO:teuthology.orchestra.run.trial043.stdout: mon reweight min pgs per osd = 4 2026-02-26T02:42:23.772 INFO:teuthology.orchestra.run.trial043.stdout: mon osd reporter subtree level = osd 2026-02-26T02:42:23.772 INFO:teuthology.orchestra.run.trial043.stdout: mon osd prime pg temp = true 2026-02-26T02:42:23.772 INFO:teuthology.orchestra.run.trial043.stdout: mon reweight min bytes per osd = 10 2026-02-26T02:42:23.772 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:23.772 INFO:teuthology.orchestra.run.trial043.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-26T02:42:23.772 INFO:teuthology.orchestra.run.trial043.stdout: auth mon ticket ttl = 660 # 11m 2026-02-26T02:42:23.772 INFO:teuthology.orchestra.run.trial043.stdout: auth service ticket ttl = 240 # 4m 2026-02-26T02:42:23.772 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:23.772 INFO:teuthology.orchestra.run.trial043.stdout: # don't complain about insecure global_id in the test suite 2026-02-26T02:42:23.772 INFO:teuthology.orchestra.run.trial043.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-26T02:42:23.772 INFO:teuthology.orchestra.run.trial043.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-26T02:42:23.772 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:23.773 INFO:teuthology.orchestra.run.trial043.stdout: # 1m isn't quite enough 2026-02-26T02:42:23.773 INFO:teuthology.orchestra.run.trial043.stdout: mon_down_mkfs_grace = 2m 2026-02-26T02:42:23.773 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:23.773 INFO:teuthology.orchestra.run.trial043.stdout: mon_warn_on_filestore_osds = false 2026-02-26T02:42:23.773 INFO:teuthology.orchestra.run.trial043.stdout: mon scrub interval = 300 2026-02-26T02:42:23.773 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:23.773 INFO:teuthology.orchestra.run.trial043.stdout:[client] 2026-02-26T02:42:23.773 INFO:teuthology.orchestra.run.trial043.stdout: rgw cache enabled = true 2026-02-26T02:42:23.773 INFO:teuthology.orchestra.run.trial043.stdout: rgw enable ops log = true 2026-02-26T02:42:23.773 INFO:teuthology.orchestra.run.trial043.stdout: rgw enable usage log = true 2026-02-26T02:42:23.773 INFO:teuthology.orchestra.run.trial043.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-26T02:42:23.773 INFO:teuthology.orchestra.run.trial043.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-26T02:42:23.773 INFO:teuthology.orchestra.run.trial043.stdout:[mon.a] 2026-02-26T02:42:23.775 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-26T02:42:23.775 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-26T02:42:23.856 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-26T02:42:23.857 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-26T02:42:23.857 DEBUG:teuthology.orchestra.run.trial043:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-26T02:42:23.912 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-26T02:42:23.912 DEBUG:teuthology.orchestra.run.trial043:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-26T02:42:23.968 INFO:tasks.ceph:Sending monmap to node ubuntu@trial043.front.sepia.ceph.com 2026-02-26T02:42:23.968 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-26T02:42:23.968 DEBUG:teuthology.orchestra.run.trial043:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-26T02:42:23.968 DEBUG:teuthology.orchestra.run.trial043:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-26T02:42:24.039 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-26T02:42:24.039 DEBUG:teuthology.orchestra.run.trial043:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-26T02:42:24.096 INFO:tasks.ceph:Setting up mon nodes... 2026-02-26T02:42:24.096 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-26T02:42:24.096 DEBUG:teuthology.orchestra.run.trial043:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-26T02:42:24.179 INFO:teuthology.orchestra.run.trial043.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-26T02:42:24.181 INFO:tasks.ceph:Setting up mds nodes... 2026-02-26T02:42:24.181 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-26T02:42:24.182 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-26T02:42:24.254 INFO:teuthology.orchestra.run.trial043.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-26T02:42:24.265 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-26T02:42:24.265 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial043.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-26T02:42:24.265 DEBUG:teuthology.orchestra.run.trial043:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-26T02:42:24.329 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T02:42:24.329 INFO:tasks.ceph:role: osd.0 2026-02-26T02:42:24.329 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial043.front.sepia.ceph.com 2026-02-26T02:42:24.330 DEBUG:teuthology.orchestra.run.trial043:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-26T02:42:24.394 INFO:teuthology.orchestra.run.trial043.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-26T02:42:24.394 INFO:teuthology.orchestra.run.trial043.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-26T02:42:24.394 INFO:teuthology.orchestra.run.trial043.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-26T02:42:24.394 INFO:teuthology.orchestra.run.trial043.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-26T02:42:24.395 INFO:teuthology.orchestra.run.trial043.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-26T02:42:24.395 INFO:teuthology.orchestra.run.trial043.stdout: = sunit=0 swidth=0 blks 2026-02-26T02:42:24.395 INFO:teuthology.orchestra.run.trial043.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-26T02:42:24.395 INFO:teuthology.orchestra.run.trial043.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-26T02:42:24.395 INFO:teuthology.orchestra.run.trial043.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-26T02:42:24.395 INFO:teuthology.orchestra.run.trial043.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-26T02:42:24.965 INFO:teuthology.orchestra.run.trial043.stdout:Discarding blocks...Done. 2026-02-26T02:42:24.967 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial043.front.sepia.ceph.com -o noatime 2026-02-26T02:42:24.967 DEBUG:teuthology.orchestra.run.trial043:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-26T02:42:25.036 DEBUG:teuthology.orchestra.run.trial043:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-26T02:42:25.103 DEBUG:teuthology.orchestra.run.trial043:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-26T02:42:25.167 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T02:42:25.167 INFO:tasks.ceph:role: osd.1 2026-02-26T02:42:25.168 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial043.front.sepia.ceph.com 2026-02-26T02:42:25.168 DEBUG:teuthology.orchestra.run.trial043:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-26T02:42:25.231 INFO:teuthology.orchestra.run.trial043.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-26T02:42:25.232 INFO:teuthology.orchestra.run.trial043.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-26T02:42:25.232 INFO:teuthology.orchestra.run.trial043.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-26T02:42:25.232 INFO:teuthology.orchestra.run.trial043.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-26T02:42:25.232 INFO:teuthology.orchestra.run.trial043.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-26T02:42:25.232 INFO:teuthology.orchestra.run.trial043.stdout: = sunit=0 swidth=0 blks 2026-02-26T02:42:25.232 INFO:teuthology.orchestra.run.trial043.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-26T02:42:25.232 INFO:teuthology.orchestra.run.trial043.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-26T02:42:25.232 INFO:teuthology.orchestra.run.trial043.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-26T02:42:25.232 INFO:teuthology.orchestra.run.trial043.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-26T02:42:25.798 INFO:teuthology.orchestra.run.trial043.stdout:Discarding blocks...Done. 2026-02-26T02:42:25.800 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial043.front.sepia.ceph.com -o noatime 2026-02-26T02:42:25.801 DEBUG:teuthology.orchestra.run.trial043:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-26T02:42:25.868 DEBUG:teuthology.orchestra.run.trial043:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-26T02:42:25.935 DEBUG:teuthology.orchestra.run.trial043:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-26T02:42:26.000 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T02:42:26.000 INFO:tasks.ceph:role: osd.2 2026-02-26T02:42:26.000 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial043.front.sepia.ceph.com 2026-02-26T02:42:26.000 DEBUG:teuthology.orchestra.run.trial043:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-26T02:42:26.064 INFO:teuthology.orchestra.run.trial043.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-26T02:42:26.065 INFO:teuthology.orchestra.run.trial043.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-26T02:42:26.065 INFO:teuthology.orchestra.run.trial043.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-26T02:42:26.065 INFO:teuthology.orchestra.run.trial043.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-26T02:42:26.065 INFO:teuthology.orchestra.run.trial043.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-26T02:42:26.065 INFO:teuthology.orchestra.run.trial043.stdout: = sunit=0 swidth=0 blks 2026-02-26T02:42:26.065 INFO:teuthology.orchestra.run.trial043.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-26T02:42:26.065 INFO:teuthology.orchestra.run.trial043.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-26T02:42:26.065 INFO:teuthology.orchestra.run.trial043.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-26T02:42:26.065 INFO:teuthology.orchestra.run.trial043.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-26T02:42:26.632 INFO:teuthology.orchestra.run.trial043.stdout:Discarding blocks...Done. 2026-02-26T02:42:26.634 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial043.front.sepia.ceph.com -o noatime 2026-02-26T02:42:26.635 DEBUG:teuthology.orchestra.run.trial043:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-26T02:42:26.701 DEBUG:teuthology.orchestra.run.trial043:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-26T02:42:26.767 DEBUG:teuthology.orchestra.run.trial043:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-26T02:42:26.847 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-26T02:42:26.845+0000 7f370b795780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2026-02-26T02:42:26.847 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-26T02:42:26.845+0000 7f370b795780 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2026-02-26T02:42:26.847 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-26T02:42:26.845+0000 7f370b795780 -1 bdev(0x555e17d49800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2026-02-26T02:42:26.847 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-26T02:42:26.845+0000 7f370b795780 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2026-02-26T02:42:30.743 DEBUG:teuthology.orchestra.run.trial043:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-26T02:42:30.807 DEBUG:teuthology.orchestra.run.trial043:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-26T02:42:30.885 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-26T02:42:30.882+0000 7fa7fc3b4780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2026-02-26T02:42:30.885 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-26T02:42:30.882+0000 7fa7fc3b4780 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2026-02-26T02:42:30.885 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-26T02:42:30.882+0000 7fa7fc3b4780 -1 bdev(0x56326a947800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2026-02-26T02:42:30.885 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-26T02:42:30.883+0000 7fa7fc3b4780 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2026-02-26T02:42:34.776 DEBUG:teuthology.orchestra.run.trial043:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-26T02:42:34.842 DEBUG:teuthology.orchestra.run.trial043:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-26T02:42:34.921 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-26T02:42:34.919+0000 7f5d2a814780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2026-02-26T02:42:34.921 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-26T02:42:34.919+0000 7f5d2a814780 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2026-02-26T02:42:34.921 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-26T02:42:34.919+0000 7f5d2a814780 -1 bdev(0x55a9620e1800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2026-02-26T02:42:34.921 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-26T02:42:34.919+0000 7f5d2a814780 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2026-02-26T02:42:38.815 DEBUG:teuthology.orchestra.run.trial043:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-26T02:42:38.880 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-26T02:42:38.880 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-26T02:42:38.880 DEBUG:teuthology.orchestra.run.trial043:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-26T02:42:38.944 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-26T02:42:38.944 DEBUG:teuthology.orchestra.run.trial043:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-26T02:42:39.006 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-26T02:42:39.006 DEBUG:teuthology.orchestra.run.trial043:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-26T02:42:39.070 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-26T02:42:39.071 DEBUG:teuthology.orchestra.run.trial043:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-26T02:42:39.133 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-26T02:42:39.134 DEBUG:teuthology.orchestra.run.trial043:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-26T02:42:39.190 INFO:tasks.ceph:Adding keys to all mons... 2026-02-26T02:42:39.190 DEBUG:teuthology.orchestra.run.trial043:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-26T02:42:39.252 INFO:teuthology.orchestra.run.trial043.stdout:[mgr.x] 2026-02-26T02:42:39.253 INFO:teuthology.orchestra.run.trial043.stdout: key = AQAQs59p4CyUChAAbbqjiJ+qZcUHxIZI1tI9hQ== 2026-02-26T02:42:39.253 INFO:teuthology.orchestra.run.trial043.stdout:[osd.0] 2026-02-26T02:42:39.253 INFO:teuthology.orchestra.run.trial043.stdout: key = AQASs59pPcRrMhAAt+UA5ytis//+fETvGfSWiw== 2026-02-26T02:42:39.253 INFO:teuthology.orchestra.run.trial043.stdout:[osd.1] 2026-02-26T02:42:39.253 INFO:teuthology.orchestra.run.trial043.stdout: key = AQAWs59p0GCjNBAAxkFXwbjhAjtlxJKaHbEMMg== 2026-02-26T02:42:39.253 INFO:teuthology.orchestra.run.trial043.stdout:[osd.2] 2026-02-26T02:42:39.253 INFO:teuthology.orchestra.run.trial043.stdout: key = AQAas59puM3SNhAAtg8kZw9x3uGW9SjJku9ORg== 2026-02-26T02:42:39.253 INFO:teuthology.orchestra.run.trial043.stdout:[client.0] 2026-02-26T02:42:39.254 INFO:teuthology.orchestra.run.trial043.stdout: key = AQAQs59psUEPDxAADDrMi0DStRP7XX5QKBidLg== 2026-02-26T02:42:39.255 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-26T02:42:39.336 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-26T02:42:39.415 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-26T02:42:39.495 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-26T02:42:39.573 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-26T02:42:39.654 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-26T02:42:39.654 DEBUG:teuthology.orchestra.run.trial043:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-26T02:42:39.718 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-26T02:42:39.801 DEBUG:teuthology.orchestra.run.trial043:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-26T02:42:39.865 DEBUG:teuthology.orchestra.run.trial043:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-26T02:42:39.921 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-26T02:42:39.921 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-26T02:42:39.922 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-26T02:42:39.963 INFO:tasks.ceph.mon.a:Started 2026-02-26T02:42:39.964 DEBUG:teuthology.orchestra.run.trial043:> sudo ceph config set mgr mgr_pool false --force 2026-02-26T02:42:40.071 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-26T02:42:40.072 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-26T02:42:40.072 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-26T02:42:40.073 INFO:tasks.ceph.mgr.x:Started 2026-02-26T02:42:40.073 INFO:tasks.ceph:Setting crush tunables to default 2026-02-26T02:42:40.073 DEBUG:teuthology.orchestra.run.trial043:> sudo ceph --cluster ceph osd crush tunables default 2026-02-26T02:42:40.172 INFO:teuthology.orchestra.run.trial043.stderr:adjusted tunables profile to default 2026-02-26T02:42:40.180 INFO:tasks.ceph:check_enable_crimson: False 2026-02-26T02:42:40.180 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-26T02:42:40.181 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-26T02:42:40.181 DEBUG:teuthology.orchestra.run.trial043:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-26T02:42:40.206 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-26T02:42:40.206 DEBUG:teuthology.orchestra.run.trial043:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-26T02:42:40.271 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-26T02:42:40.271 DEBUG:teuthology.orchestra.run.trial043:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-26T02:42:40.336 DEBUG:teuthology.orchestra.run.trial043:> sudo ceph --cluster ceph osd new 3383d670-1454-4abb-8847-9dab847e0a5a 0 2026-02-26T02:42:40.479 INFO:teuthology.orchestra.run.trial043.stdout:0 2026-02-26T02:42:40.486 INFO:tasks.ceph.mgr.x.trial043.stderr:2026-02-26T02:42:40.483+0000 7efc0d02d140 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2026-02-26T02:42:40.488 DEBUG:teuthology.orchestra.run.trial043:> sudo ceph --cluster ceph osd new 729a8cd6-7014-40b4-9e02-940e867460f1 1 2026-02-26T02:42:40.591 INFO:teuthology.orchestra.run.trial043.stdout:1 2026-02-26T02:42:40.599 DEBUG:teuthology.orchestra.run.trial043:> sudo ceph --cluster ceph osd new b1591962-eecc-4a4f-83d1-040bee63ba29 2 2026-02-26T02:42:40.609 INFO:tasks.ceph.mgr.x.trial043.stderr:2026-02-26T02:42:40.607+0000 7efc0d02d140 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2026-02-26T02:42:40.639 INFO:tasks.ceph.mgr.x.trial043.stderr:2026-02-26T02:42:40.637+0000 7efc0d02d140 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2026-02-26T02:42:40.673 INFO:tasks.ceph.mgr.x.trial043.stderr:2026-02-26T02:42:40.671+0000 7efc0d02d140 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2026-02-26T02:42:40.697 INFO:tasks.ceph.mgr.x.trial043.stderr:2026-02-26T02:42:40.695+0000 7efc0d02d140 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2026-02-26T02:42:40.698 INFO:teuthology.orchestra.run.trial043.stdout:2 2026-02-26T02:42:40.706 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-26T02:42:40.706 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2026-02-26T02:42:40.724 INFO:tasks.ceph.mgr.x.trial043.stderr:2026-02-26T02:42:40.722+0000 7efc0d02d140 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2026-02-26T02:42:40.725 INFO:tasks.ceph.osd.0:Started 2026-02-26T02:42:40.726 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-26T02:42:40.726 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2026-02-26T02:42:40.727 INFO:tasks.ceph.osd.1:Started 2026-02-26T02:42:40.727 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-26T02:42:40.727 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2026-02-26T02:42:40.728 INFO:tasks.ceph.osd.2:Started 2026-02-26T02:42:40.729 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T02:42:40.784 INFO:tasks.ceph.mgr.x.trial043.stderr:2026-02-26T02:42:40.782+0000 7efc0d02d140 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2026-02-26T02:42:40.828 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:42:40.828 INFO:teuthology.orchestra.run.trial043.stdout:{"epoch":5,"fsid":"6dea32ae-afd8-4e41-a06e-2ec4e4797235","created":"2026-02-26T02:42:40.023737+0000","modified":"2026-02-26T02:42:40.696050+0000","last_up_change":"0.000000","last_in_change":"2026-02-26T02:42:40.696050+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383d670-1454-4abb-8847-9dab847e0a5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"729a8cd6-7014-40b4-9e02-940e867460f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b1591962-eecc-4a4f-83d1-040bee63ba29","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T02:42:40.834 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-26T02:42:40.835 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-26T02:42:40.850 INFO:tasks.ceph.mgr.x.trial043.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-26T02:42:40.850 INFO:tasks.ceph.mgr.x.trial043.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-26T02:42:40.850 INFO:tasks.ceph.mgr.x.trial043.stderr: from numpy import show_config as show_numpy_config 2026-02-26T02:42:40.853 INFO:tasks.ceph.mgr.x.trial043.stderr:2026-02-26T02:42:40.852+0000 7efc0d02d140 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2026-02-26T02:42:40.879 INFO:tasks.ceph.mgr.x.trial043.stderr:2026-02-26T02:42:40.877+0000 7efc0d02d140 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2026-02-26T02:42:40.935 INFO:tasks.ceph.mgr.x.trial043.stderr:2026-02-26T02:42:40.933+0000 7efc0d02d140 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2026-02-26T02:42:40.978 INFO:tasks.ceph.mgr.x.trial043.stderr:2026-02-26T02:42:40.976+0000 7efc0d02d140 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2026-02-26T02:42:41.004 INFO:tasks.ceph.mgr.x.trial043.stderr:2026-02-26T02:42:41.002+0000 7efc0d02d140 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2026-02-26T02:42:41.028 INFO:tasks.ceph.mgr.x.trial043.stderr:2026-02-26T02:42:41.026+0000 7efc0d02d140 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2026-02-26T02:42:41.054 INFO:tasks.ceph.mgr.x.trial043.stderr:2026-02-26T02:42:41.052+0000 7efc0d02d140 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2026-02-26T02:42:41.077 INFO:tasks.ceph.mgr.x.trial043.stderr:2026-02-26T02:42:41.075+0000 7efc0d02d140 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2026-02-26T02:42:41.137 DEBUG:teuthology.orchestra.run.trial043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-26T02:42:41.228 INFO:teuthology.misc.health.trial043.stdout: 2026-02-26T02:42:41.228 INFO:teuthology.misc.health.trial043.stdout:{"epoch":5,"fsid":"6dea32ae-afd8-4e41-a06e-2ec4e4797235","created":"2026-02-26T02:42:40.023737+0000","modified":"2026-02-26T02:42:40.696050+0000","last_up_change":"0.000000","last_in_change":"2026-02-26T02:42:40.696050+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383d670-1454-4abb-8847-9dab847e0a5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"729a8cd6-7014-40b4-9e02-940e867460f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b1591962-eecc-4a4f-83d1-040bee63ba29","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T02:42:41.235 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-26T02:42:41.277 INFO:tasks.ceph.mgr.x.trial043.stderr:2026-02-26T02:42:41.275+0000 7efc0d02d140 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2026-02-26T02:42:41.332 INFO:tasks.ceph.mgr.x.trial043.stderr:2026-02-26T02:42:41.330+0000 7efc0d02d140 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2026-02-26T02:42:41.354 INFO:tasks.ceph.mgr.x.trial043.stderr:2026-02-26T02:42:41.352+0000 7efc0d02d140 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2026-02-26T02:42:41.377 INFO:tasks.ceph.mgr.x.trial043.stderr:2026-02-26T02:42:41.375+0000 7efc0d02d140 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2026-02-26T02:42:41.430 INFO:tasks.ceph.mgr.x.trial043.stderr:2026-02-26T02:42:41.428+0000 7efc0d02d140 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2026-02-26T02:42:41.451 INFO:tasks.ceph.mgr.x.trial043.stderr:2026-02-26T02:42:41.449+0000 7efc0d02d140 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2026-02-26T02:42:41.503 INFO:tasks.ceph.mgr.x.trial043.stderr:2026-02-26T02:42:41.501+0000 7efc0d02d140 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2026-02-26T02:42:41.902 INFO:tasks.ceph.mgr.x.trial043.stderr:2026-02-26T02:42:41.900+0000 7efc0d02d140 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2026-02-26T02:42:41.924 INFO:tasks.ceph.mgr.x.trial043.stderr:2026-02-26T02:42:41.922+0000 7efc0d02d140 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2026-02-26T02:42:42.268 INFO:tasks.ceph.mgr.x.trial043.stderr:2026-02-26T02:42:42.266+0000 7efc0d02d140 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2026-02-26T02:42:42.656 INFO:tasks.ceph.osd.2.trial043.stderr:2026-02-26T02:42:42.653+0000 7faecce11780 -1 Falling back to public interface 2026-02-26T02:42:42.660 INFO:tasks.ceph.osd.0.trial043.stderr:2026-02-26T02:42:42.657+0000 7ff7a2b34780 -1 Falling back to public interface 2026-02-26T02:42:42.660 INFO:tasks.ceph.osd.1.trial043.stderr:2026-02-26T02:42:42.657+0000 7f7ae8bf5780 -1 Falling back to public interface 2026-02-26T02:42:43.274 INFO:tasks.ceph.mgr.x.trial043.stderr:2026-02-26T02:42:43.272+0000 7efb2958e640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-26T02:42:47.173 INFO:tasks.ceph.osd.2.trial043.stderr:2026-02-26T02:42:47.170+0000 7faecce11780 -1 osd.2 0 log_to_monitors true 2026-02-26T02:42:47.215 INFO:tasks.ceph.osd.0.trial043.stderr:2026-02-26T02:42:47.212+0000 7ff7a2b34780 -1 osd.0 0 log_to_monitors true 2026-02-26T02:42:47.255 INFO:tasks.ceph.osd.1.trial043.stderr:2026-02-26T02:42:47.252+0000 7f7ae8bf5780 -1 osd.1 0 log_to_monitors true 2026-02-26T02:42:47.537 DEBUG:teuthology.orchestra.run.trial043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-26T02:42:47.686 INFO:teuthology.misc.health.trial043.stdout: 2026-02-26T02:42:47.687 INFO:teuthology.misc.health.trial043.stdout:{"epoch":6,"fsid":"6dea32ae-afd8-4e41-a06e-2ec4e4797235","created":"2026-02-26T02:42:40.023737+0000","modified":"2026-02-26T02:42:47.224022+0000","last_up_change":"0.000000","last_in_change":"2026-02-26T02:42:40.696050+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383d670-1454-4abb-8847-9dab847e0a5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"729a8cd6-7014-40b4-9e02-940e867460f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b1591962-eecc-4a4f-83d1-040bee63ba29","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T02:42:47.693 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-26T02:42:48.233 INFO:tasks.ceph.osd.2.trial043.stderr:2026-02-26T02:42:48.230+0000 7faec8016640 -1 osd.2 0 waiting for initial osdmap 2026-02-26T02:42:48.234 INFO:tasks.ceph.osd.2.trial043.stderr:2026-02-26T02:42:48.231+0000 7faec4652640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-26T02:42:48.494 INFO:tasks.ceph.osd.0.trial043.stderr:2026-02-26T02:42:48.491+0000 7ff79e29e640 -1 osd.0 0 waiting for initial osdmap 2026-02-26T02:42:48.497 INFO:tasks.ceph.osd.0.trial043.stderr:2026-02-26T02:42:48.494+0000 7ff7968d2640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-26T02:42:49.231 INFO:tasks.ceph.osd.1.trial043.stderr:2026-02-26T02:42:49.228+0000 7f7ae435f640 -1 osd.1 0 waiting for initial osdmap 2026-02-26T02:42:49.233 INFO:tasks.ceph.osd.1.trial043.stderr:2026-02-26T02:42:49.230+0000 7f7ae099b640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-26T02:42:50.179 INFO:tasks.ceph.osd.2.trial043.stderr:2026-02-26T02:42:50.176+0000 7faebf058640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-26T02:42:53.996 DEBUG:teuthology.orchestra.run.trial043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-26T02:42:54.143 INFO:teuthology.misc.health.trial043.stdout: 2026-02-26T02:42:54.144 INFO:teuthology.misc.health.trial043.stdout:{"epoch":9,"fsid":"6dea32ae-afd8-4e41-a06e-2ec4e4797235","created":"2026-02-26T02:42:40.023737+0000","modified":"2026-02-26T02:42:50.231670+0000","last_up_change":"2026-02-26T02:42:50.231670+0000","last_in_change":"2026-02-26T02:42:40.696050+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3383d670-1454-4abb-8847-9dab847e0a5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6805","nonce":1923240457}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6807","nonce":1923240457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6812","nonce":1923240457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6809","nonce":1923240457}]},"public_addr":"10.20.193.43:6805/1923240457","cluster_addr":"10.20.193.43:6807/1923240457","heartbeat_back_addr":"10.20.193.43:6812/1923240457","heartbeat_front_addr":"10.20.193.43:6809/1923240457","state":["exists","up"]},{"osd":1,"uuid":"729a8cd6-7014-40b4-9e02-940e867460f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6806","nonce":4263071973}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6808","nonce":4263071973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6811","nonce":4263071973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6810","nonce":4263071973}]},"public_addr":"10.20.193.43:6806/4263071973","cluster_addr":"10.20.193.43:6808/4263071973","heartbeat_back_addr":"10.20.193.43:6811/4263071973","heartbeat_front_addr":"10.20.193.43:6810/4263071973","state":["exists","up"]},{"osd":2,"uuid":"b1591962-eecc-4a4f-83d1-040bee63ba29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6801","nonce":1886150811}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6802","nonce":1886150811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6804","nonce":1886150811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6803","nonce":1886150811}]},"public_addr":"10.20.193.43:6801/1886150811","cluster_addr":"10.20.193.43:6802/1886150811","heartbeat_back_addr":"10.20.193.43:6804/1886150811","heartbeat_front_addr":"10.20.193.43:6803/1886150811","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T02:42:54.149 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-26T02:42:54.150 INFO:tasks.ceph:Creating RBD pool 2026-02-26T02:42:54.150 DEBUG:teuthology.orchestra.run.trial043:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-26T02:42:54.325 INFO:teuthology.orchestra.run.trial043.stderr:pool 'rbd' created 2026-02-26T02:42:54.334 DEBUG:teuthology.orchestra.run.trial043:> rbd --cluster ceph pool init rbd 2026-02-26T02:43:00.340 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-26T02:43:00.340 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-26T02:43:00.340 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-26T02:43:00.340 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-26T02:43:00.340 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-26T02:43:00.519 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:43:00.526 INFO:teuthology.orchestra.run.trial043.stdout:{"epoch":4,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6800","nonce":3619129573}]},"active_addr":"10.20.193.43:6800/3619129573","active_change":"2026-02-26T02:42:42.267817+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number. if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.43:0","nonce":3642507123}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.43:0","nonce":2134978891}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.43:0","nonce":3371346229}]}]} 2026-02-26T02:43:00.531 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-26T02:43:00.532 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-26T02:43:00.532 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T02:43:00.686 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:43:00.686 INFO:teuthology.orchestra.run.trial043.stdout:{"epoch":15,"fsid":"6dea32ae-afd8-4e41-a06e-2ec4e4797235","created":"2026-02-26T02:42:40.023737+0000","modified":"2026-02-26T02:43:00.331390+0000","last_up_change":"2026-02-26T02:42:50.231670+0000","last_in_change":"2026-02-26T02:42:40.696050+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-26T02:42:54.301050+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3383d670-1454-4abb-8847-9dab847e0a5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6805","nonce":1923240457}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6807","nonce":1923240457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6812","nonce":1923240457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6809","nonce":1923240457}]},"public_addr":"10.20.193.43:6805/1923240457","cluster_addr":"10.20.193.43:6807/1923240457","heartbeat_back_addr":"10.20.193.43:6812/1923240457","heartbeat_front_addr":"10.20.193.43:6809/1923240457","state":["exists","up"]},{"osd":1,"uuid":"729a8cd6-7014-40b4-9e02-940e867460f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6806","nonce":4263071973}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6808","nonce":4263071973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6811","nonce":4263071973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6810","nonce":4263071973}]},"public_addr":"10.20.193.43:6806/4263071973","cluster_addr":"10.20.193.43:6808/4263071973","heartbeat_back_addr":"10.20.193.43:6811/4263071973","heartbeat_front_addr":"10.20.193.43:6810/4263071973","state":["exists","up"]},{"osd":2,"uuid":"b1591962-eecc-4a4f-83d1-040bee63ba29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6801","nonce":1886150811}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6802","nonce":1886150811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6804","nonce":1886150811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6803","nonce":1886150811}]},"public_addr":"10.20.193.43:6801/1886150811","cluster_addr":"10.20.193.43:6802/1886150811","heartbeat_back_addr":"10.20.193.43:6804/1886150811","heartbeat_front_addr":"10.20.193.43:6803/1886150811","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T02:42:48.220681+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T02:42:48.293491+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T02:42:48.146728+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T02:43:00.694 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-26T02:43:00.694 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T02:43:00.854 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:43:00.854 INFO:teuthology.orchestra.run.trial043.stdout:{"epoch":15,"fsid":"6dea32ae-afd8-4e41-a06e-2ec4e4797235","created":"2026-02-26T02:42:40.023737+0000","modified":"2026-02-26T02:43:00.331390+0000","last_up_change":"2026-02-26T02:42:50.231670+0000","last_in_change":"2026-02-26T02:42:40.696050+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-26T02:42:54.301050+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3383d670-1454-4abb-8847-9dab847e0a5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6805","nonce":1923240457}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6807","nonce":1923240457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6812","nonce":1923240457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6809","nonce":1923240457}]},"public_addr":"10.20.193.43:6805/1923240457","cluster_addr":"10.20.193.43:6807/1923240457","heartbeat_back_addr":"10.20.193.43:6812/1923240457","heartbeat_front_addr":"10.20.193.43:6809/1923240457","state":["exists","up"]},{"osd":1,"uuid":"729a8cd6-7014-40b4-9e02-940e867460f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6806","nonce":4263071973}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6808","nonce":4263071973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6811","nonce":4263071973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6810","nonce":4263071973}]},"public_addr":"10.20.193.43:6806/4263071973","cluster_addr":"10.20.193.43:6808/4263071973","heartbeat_back_addr":"10.20.193.43:6811/4263071973","heartbeat_front_addr":"10.20.193.43:6810/4263071973","state":["exists","up"]},{"osd":2,"uuid":"b1591962-eecc-4a4f-83d1-040bee63ba29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6801","nonce":1886150811}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6802","nonce":1886150811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6804","nonce":1886150811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6803","nonce":1886150811}]},"public_addr":"10.20.193.43:6801/1886150811","cluster_addr":"10.20.193.43:6802/1886150811","heartbeat_back_addr":"10.20.193.43:6804/1886150811","heartbeat_front_addr":"10.20.193.43:6803/1886150811","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T02:42:48.220681+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T02:42:48.293491+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T02:42:48.146728+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T02:43:00.864 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-26T02:43:00.864 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-26T02:43:00.865 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-26T02:43:00.951 INFO:teuthology.orchestra.run.trial043.stdout:34359738372 2026-02-26T02:43:00.951 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-26T02:43:00.951 INFO:teuthology.orchestra.run.trial043.stdout:34359738372 2026-02-26T02:43:00.952 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-26T02:43:00.952 INFO:teuthology.orchestra.run.trial043.stdout:38654705669 2026-02-26T02:43:00.952 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-26T02:43:01.149 INFO:teuthology.orchestra.run.trial043.stdout:38654705668 2026-02-26T02:43:01.153 INFO:teuthology.orchestra.run.trial043.stdout:34359738371 2026-02-26T02:43:01.153 INFO:teuthology.orchestra.run.trial043.stdout:34359738371 2026-02-26T02:43:01.159 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705668 for osd.1 2026-02-26T02:43:01.163 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.0 2026-02-26T02:43:01.163 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.2 2026-02-26T02:43:02.160 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-26T02:43:02.163 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-26T02:43:02.164 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-26T02:43:02.320 INFO:teuthology.orchestra.run.trial043.stdout:38654705668 2026-02-26T02:43:02.330 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705668 for osd.1 2026-02-26T02:43:02.362 INFO:teuthology.orchestra.run.trial043.stdout:34359738372 2026-02-26T02:43:02.364 INFO:teuthology.orchestra.run.trial043.stdout:34359738372 2026-02-26T02:43:02.372 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.2 2026-02-26T02:43:02.372 DEBUG:teuthology.parallel:result is None 2026-02-26T02:43:02.375 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.0 2026-02-26T02:43:02.375 DEBUG:teuthology.parallel:result is None 2026-02-26T02:43:03.331 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-26T02:43:03.488 INFO:teuthology.orchestra.run.trial043.stdout:38654705670 2026-02-26T02:43:03.497 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705670 for osd.1 2026-02-26T02:43:03.498 DEBUG:teuthology.parallel:result is None 2026-02-26T02:43:03.498 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-26T02:43:03.498 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T02:43:03.652 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:43:03.653 INFO:teuthology.orchestra.run.trial043.stderr:dumped all 2026-02-26T02:43:03.659 INFO:teuthology.orchestra.run.trial043.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2026-02-26T02:43:02.271785+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81272,"kb_used_data":368,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":283034248,"statfs":{"total":289910292480,"available":289827069952,"internally_reserved":0,"allocated":376832,"data_stored":88170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82767186},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":25,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-26T02:43:00.335058+0000","last_change":"2026-02-26T02:43:00.335177+0000","last_active":"2026-02-26T02:43:00.335058+0000","last_peered":"2026-02-26T02:43:00.335058+0000","last_clean":"2026-02-26T02:43:00.335058+0000","last_became_active":"2026-02-26T02:42:55.329119+0000","last_became_peered":"2026-02-26T02:42:55.329119+0000","last_unstale":"2026-02-26T02:43:00.335058+0000","last_undegraded":"2026-02-26T02:43:00.335058+0000","last_fullsized":"2026-02-26T02:43:00.335058+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T09:46:23.963835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047321700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":25,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-26T02:43:00.335162+0000","last_change":"2026-02-26T02:43:00.335304+0000","last_active":"2026-02-26T02:43:00.335162+0000","last_peered":"2026-02-26T02:43:00.335162+0000","last_clean":"2026-02-26T02:43:00.335162+0000","last_became_active":"2026-02-26T02:42:55.329558+0000","last_became_peered":"2026-02-26T02:42:55.329558+0000","last_unstale":"2026-02-26T02:43:00.335162+0000","last_undegraded":"2026-02-26T02:43:00.335162+0000","last_fullsized":"2026-02-26T02:43:00.335162+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T09:22:03.395212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055368699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-26T02:42:58.332390+0000","last_change":"2026-02-26T02:42:55.329636+0000","last_active":"2026-02-26T02:42:58.332390+0000","last_peered":"2026-02-26T02:42:58.332390+0000","last_clean":"2026-02-26T02:42:58.332390+0000","last_became_active":"2026-02-26T02:42:55.329459+0000","last_became_peered":"2026-02-26T02:42:55.329459+0000","last_unstale":"2026-02-26T02:42:58.332390+0000","last_undegraded":"2026-02-26T02:42:58.332390+0000","last_fullsized":"2026-02-26T02:42:58.332390+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T04:12:39.722388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":25,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-26T02:43:00.335023+0000","last_change":"2026-02-26T02:43:00.335133+0000","last_active":"2026-02-26T02:43:00.335023+0000","last_peered":"2026-02-26T02:43:00.335023+0000","last_clean":"2026-02-26T02:43:00.335023+0000","last_became_active":"2026-02-26T02:42:55.329615+0000","last_became_peered":"2026-02-26T02:42:55.329615+0000","last_unstale":"2026-02-26T02:43:00.335023+0000","last_undegraded":"2026-02-26T02:43:00.335023+0000","last_fullsized":"2026-02-26T02:43:00.335023+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T08:01:46.770260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044036699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.3","version":"13'1","reported_seq":26,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-26T02:43:00.334882+0000","last_change":"2026-02-26T02:43:00.335016+0000","last_active":"2026-02-26T02:43:00.334882+0000","last_peered":"2026-02-26T02:43:00.334882+0000","last_clean":"2026-02-26T02:43:00.334882+0000","last_became_active":"2026-02-26T02:42:55.329117+0000","last_became_peered":"2026-02-26T02:42:55.329117+0000","last_unstale":"2026-02-26T02:43:00.334882+0000","last_undegraded":"2026-02-26T02:43:00.334882+0000","last_fullsized":"2026-02-26T02:43:00.334882+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T07:13:37.435598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035731199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"15'2","reported_seq":27,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-26T02:43:00.333293+0000","last_change":"2026-02-26T02:43:00.333293+0000","last_active":"2026-02-26T02:43:00.333293+0000","last_peered":"2026-02-26T02:43:00.333293+0000","last_clean":"2026-02-26T02:43:00.333293+0000","last_became_active":"2026-02-26T02:42:55.329420+0000","last_became_peered":"2026-02-26T02:42:55.329420+0000","last_unstale":"2026-02-26T02:43:00.333293+0000","last_undegraded":"2026-02-26T02:43:00.333293+0000","last_fullsized":"2026-02-26T02:43:00.333293+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T03:27:58.196416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027048099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-26T02:42:58.332403+0000","last_change":"2026-02-26T02:42:55.329725+0000","last_active":"2026-02-26T02:42:58.332403+0000","last_peered":"2026-02-26T02:42:58.332403+0000","last_clean":"2026-02-26T02:42:58.332403+0000","last_became_active":"2026-02-26T02:42:55.329517+0000","last_became_peered":"2026-02-26T02:42:55.329517+0000","last_unstale":"2026-02-26T02:42:58.332403+0000","last_undegraded":"2026-02-26T02:42:58.332403+0000","last_fullsized":"2026-02-26T02:42:58.332403+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T13:16:39.092277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":25,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-26T02:43:00.334834+0000","last_change":"2026-02-26T02:43:00.335001+0000","last_active":"2026-02-26T02:43:00.334834+0000","last_peered":"2026-02-26T02:43:00.334834+0000","last_clean":"2026-02-26T02:43:00.334834+0000","last_became_active":"2026-02-26T02:42:55.329494+0000","last_became_peered":"2026-02-26T02:42:55.329494+0000","last_unstale":"2026-02-26T02:43:00.334834+0000","last_undegraded":"2026-02-26T02:43:00.334834+0000","last_fullsized":"2026-02-26T02:43:00.334834+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T11:16:00.768429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036294300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27100,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344740,"statfs":{"total":96636764160,"available":96609013760,"internally_reserved":0,"allocated":135168,"data_stored":32226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27080,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344760,"statfs":{"total":96636764160,"available":96609034240,"internally_reserved":0,"allocated":114688,"data_stored":26633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27092,"kb_used_data":124,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344748,"statfs":{"total":96636764160,"available":96609021952,"internally_reserved":0,"allocated":126976,"data_stored":29311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T02:43:03.661 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T02:43:03.813 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:43:03.813 INFO:teuthology.orchestra.run.trial043.stderr:dumped all 2026-02-26T02:43:03.821 INFO:teuthology.orchestra.run.trial043.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2026-02-26T02:43:02.271785+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81272,"kb_used_data":368,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":283034248,"statfs":{"total":289910292480,"available":289827069952,"internally_reserved":0,"allocated":376832,"data_stored":88170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82767186},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":25,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-26T02:43:00.335058+0000","last_change":"2026-02-26T02:43:00.335177+0000","last_active":"2026-02-26T02:43:00.335058+0000","last_peered":"2026-02-26T02:43:00.335058+0000","last_clean":"2026-02-26T02:43:00.335058+0000","last_became_active":"2026-02-26T02:42:55.329119+0000","last_became_peered":"2026-02-26T02:42:55.329119+0000","last_unstale":"2026-02-26T02:43:00.335058+0000","last_undegraded":"2026-02-26T02:43:00.335058+0000","last_fullsized":"2026-02-26T02:43:00.335058+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T09:46:23.963835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047321700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":25,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-26T02:43:00.335162+0000","last_change":"2026-02-26T02:43:00.335304+0000","last_active":"2026-02-26T02:43:00.335162+0000","last_peered":"2026-02-26T02:43:00.335162+0000","last_clean":"2026-02-26T02:43:00.335162+0000","last_became_active":"2026-02-26T02:42:55.329558+0000","last_became_peered":"2026-02-26T02:42:55.329558+0000","last_unstale":"2026-02-26T02:43:00.335162+0000","last_undegraded":"2026-02-26T02:43:00.335162+0000","last_fullsized":"2026-02-26T02:43:00.335162+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T09:22:03.395212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055368699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-26T02:42:58.332390+0000","last_change":"2026-02-26T02:42:55.329636+0000","last_active":"2026-02-26T02:42:58.332390+0000","last_peered":"2026-02-26T02:42:58.332390+0000","last_clean":"2026-02-26T02:42:58.332390+0000","last_became_active":"2026-02-26T02:42:55.329459+0000","last_became_peered":"2026-02-26T02:42:55.329459+0000","last_unstale":"2026-02-26T02:42:58.332390+0000","last_undegraded":"2026-02-26T02:42:58.332390+0000","last_fullsized":"2026-02-26T02:42:58.332390+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T04:12:39.722388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":25,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-26T02:43:00.335023+0000","last_change":"2026-02-26T02:43:00.335133+0000","last_active":"2026-02-26T02:43:00.335023+0000","last_peered":"2026-02-26T02:43:00.335023+0000","last_clean":"2026-02-26T02:43:00.335023+0000","last_became_active":"2026-02-26T02:42:55.329615+0000","last_became_peered":"2026-02-26T02:42:55.329615+0000","last_unstale":"2026-02-26T02:43:00.335023+0000","last_undegraded":"2026-02-26T02:43:00.335023+0000","last_fullsized":"2026-02-26T02:43:00.335023+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T08:01:46.770260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044036699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.3","version":"13'1","reported_seq":26,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-26T02:43:00.334882+0000","last_change":"2026-02-26T02:43:00.335016+0000","last_active":"2026-02-26T02:43:00.334882+0000","last_peered":"2026-02-26T02:43:00.334882+0000","last_clean":"2026-02-26T02:43:00.334882+0000","last_became_active":"2026-02-26T02:42:55.329117+0000","last_became_peered":"2026-02-26T02:42:55.329117+0000","last_unstale":"2026-02-26T02:43:00.334882+0000","last_undegraded":"2026-02-26T02:43:00.334882+0000","last_fullsized":"2026-02-26T02:43:00.334882+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T07:13:37.435598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035731199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"15'2","reported_seq":27,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-26T02:43:00.333293+0000","last_change":"2026-02-26T02:43:00.333293+0000","last_active":"2026-02-26T02:43:00.333293+0000","last_peered":"2026-02-26T02:43:00.333293+0000","last_clean":"2026-02-26T02:43:00.333293+0000","last_became_active":"2026-02-26T02:42:55.329420+0000","last_became_peered":"2026-02-26T02:42:55.329420+0000","last_unstale":"2026-02-26T02:43:00.333293+0000","last_undegraded":"2026-02-26T02:43:00.333293+0000","last_fullsized":"2026-02-26T02:43:00.333293+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T03:27:58.196416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027048099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-26T02:42:58.332403+0000","last_change":"2026-02-26T02:42:55.329725+0000","last_active":"2026-02-26T02:42:58.332403+0000","last_peered":"2026-02-26T02:42:58.332403+0000","last_clean":"2026-02-26T02:42:58.332403+0000","last_became_active":"2026-02-26T02:42:55.329517+0000","last_became_peered":"2026-02-26T02:42:55.329517+0000","last_unstale":"2026-02-26T02:42:58.332403+0000","last_undegraded":"2026-02-26T02:42:58.332403+0000","last_fullsized":"2026-02-26T02:42:58.332403+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T13:16:39.092277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":25,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-26T02:43:00.334834+0000","last_change":"2026-02-26T02:43:00.335001+0000","last_active":"2026-02-26T02:43:00.334834+0000","last_peered":"2026-02-26T02:43:00.334834+0000","last_clean":"2026-02-26T02:43:00.334834+0000","last_became_active":"2026-02-26T02:42:55.329494+0000","last_became_peered":"2026-02-26T02:42:55.329494+0000","last_unstale":"2026-02-26T02:43:00.334834+0000","last_undegraded":"2026-02-26T02:43:00.334834+0000","last_fullsized":"2026-02-26T02:43:00.334834+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T11:16:00.768429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036294300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27100,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344740,"statfs":{"total":96636764160,"available":96609013760,"internally_reserved":0,"allocated":135168,"data_stored":32226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27080,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344760,"statfs":{"total":96636764160,"available":96609034240,"internally_reserved":0,"allocated":114688,"data_stored":26633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27092,"kb_used_data":124,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344748,"statfs":{"total":96636764160,"available":96609021952,"internally_reserved":0,"allocated":126976,"data_stored":29311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T02:43:03.821 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-26T02:43:03.821 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-26T02:43:03.821 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-26T02:43:03.821 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-26T02:43:03.989 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:43:03.989 INFO:teuthology.orchestra.run.trial043.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-26T02:43:03.998 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-26T02:43:03.998 INFO:teuthology.run_tasks:Running task workunit... 2026-02-26T02:43:04.027 INFO:tasks.workunit:Pulling workunits from ref 8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5 2026-02-26T02:43:04.028 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-26T02:43:04.028 DEBUG:teuthology.orchestra.run.trial043:> stat -- /home/ubuntu/cephtest/mnt.0 2026-02-26T02:43:04.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T02:43:04.045 INFO:teuthology.orchestra.run.trial043.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2026-02-26T02:43:04.045 DEBUG:teuthology.orchestra.run.trial043:> mkdir -- /home/ubuntu/cephtest/mnt.0 2026-02-26T02:43:04.102 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2026-02-26T02:43:04.102 DEBUG:teuthology.orchestra.run.trial043:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2026-02-26T02:43:04.158 INFO:tasks.workunit:timeout=3h 2026-02-26T02:43:04.158 INFO:tasks.workunit:cleanup=True 2026-02-26T02:43:04.158 DEBUG:teuthology.orchestra.run.trial043:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5 2026-02-26T02:43:04.220 INFO:tasks.workunit.client.0.trial043.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2026-02-26T02:43:24.701 INFO:tasks.workunit.client.0.trial043.stderr:Note: switching to '8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5'. 2026-02-26T02:43:24.701 INFO:tasks.workunit.client.0.trial043.stderr: 2026-02-26T02:43:24.702 INFO:tasks.workunit.client.0.trial043.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-26T02:43:24.702 INFO:tasks.workunit.client.0.trial043.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-26T02:43:24.702 INFO:tasks.workunit.client.0.trial043.stderr:state without impacting any branches by switching back to a branch. 2026-02-26T02:43:24.702 INFO:tasks.workunit.client.0.trial043.stderr: 2026-02-26T02:43:24.702 INFO:tasks.workunit.client.0.trial043.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-26T02:43:24.702 INFO:tasks.workunit.client.0.trial043.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-26T02:43:24.702 INFO:tasks.workunit.client.0.trial043.stderr: 2026-02-26T02:43:24.702 INFO:tasks.workunit.client.0.trial043.stderr: git switch -c 2026-02-26T02:43:24.702 INFO:tasks.workunit.client.0.trial043.stderr: 2026-02-26T02:43:24.703 INFO:tasks.workunit.client.0.trial043.stderr:Or undo this operation with: 2026-02-26T02:43:24.703 INFO:tasks.workunit.client.0.trial043.stderr: 2026-02-26T02:43:24.703 INFO:tasks.workunit.client.0.trial043.stderr: git switch - 2026-02-26T02:43:24.703 INFO:tasks.workunit.client.0.trial043.stderr: 2026-02-26T02:43:24.703 INFO:tasks.workunit.client.0.trial043.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-26T02:43:24.703 INFO:tasks.workunit.client.0.trial043.stderr: 2026-02-26T02:43:24.703 INFO:tasks.workunit.client.0.trial043.stderr:HEAD is now at 8d724284d6d Merge branch 'wip-ifed-fix-vselector-math-squi' of https://github.com/ifed01/ceph into wip-yuri9-testing-2026-02-25-1600-squid 2026-02-26T02:43:24.709 DEBUG:teuthology.orchestra.run.trial043:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2026-02-26T02:43:24.765 INFO:tasks.workunit.client.0.trial043.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2026-02-26T02:43:24.767 INFO:tasks.workunit.client.0.trial043.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-26T02:43:24.767 INFO:tasks.workunit.client.0.trial043.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2026-02-26T02:43:24.854 INFO:tasks.workunit.client.0.trial043.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2026-02-26T02:43:24.884 INFO:tasks.workunit.client.0.trial043.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2026-02-26T02:43:24.909 INFO:tasks.workunit.client.0.trial043.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-26T02:43:24.911 INFO:tasks.workunit.client.0.trial043.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-26T02:43:24.911 INFO:tasks.workunit.client.0.trial043.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2026-02-26T02:43:24.934 INFO:tasks.workunit.client.0.trial043.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-26T02:43:24.938 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-26T02:43:24.938 DEBUG:teuthology.orchestra.run.trial043:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2026-02-26T02:43:24.994 INFO:tasks.workunit:Running workunits matching rados/test_crash.sh on client.0... 2026-02-26T02:43:24.995 INFO:tasks.workunit:Running workunit rados/test_crash.sh... 2026-02-26T02:43:24.995 DEBUG:teuthology.orchestra.run.trial043:workunit test rados/test_crash.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_crash.sh 2026-02-26T02:43:25.053 INFO:tasks.workunit.client.0.trial043.stderr:+ sudo killall -ABRT ceph-osd 2026-02-26T02:43:25.076 INFO:tasks.ceph.osd.0.trial043.stderr:*** Caught signal (Aborted) ** 2026-02-26T02:43:25.076 INFO:tasks.ceph.osd.0.trial043.stderr: in thread 7ff7a2b34780 thread_name:ceph-osd 2026-02-26T02:43:25.076 INFO:tasks.ceph.osd.1.trial043.stderr:*** Caught signal (Aborted) ** 2026-02-26T02:43:25.076 INFO:tasks.ceph.osd.1.trial043.stderr: in thread 7f7ae8bf5780 thread_name:ceph-osd 2026-02-26T02:43:25.077 INFO:tasks.ceph.osd.2.trial043.stderr:*** Caught signal (Aborted) ** 2026-02-26T02:43:25.077 INFO:tasks.ceph.osd.2.trial043.stderr: in thread 7faecce11780 thread_name:ceph-osd 2026-02-26T02:43:25.077 INFO:tasks.ceph.osd.2.trial043.stderr: ceph version 19.2.3-687-g8d724284 (8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5) squid (stable) 2026-02-26T02:43:25.077 INFO:tasks.ceph.osd.2.trial043.stderr: 1: /lib64/libc.so.6(+0x3fc30) [0x7faeccc3fc30] 2026-02-26T02:43:25.077 INFO:tasks.ceph.osd.2.trial043.stderr: 2: /lib64/libc.so.6(+0x8838a) [0x7faeccc8838a] 2026-02-26T02:43:25.077 INFO:tasks.ceph.osd.2.trial043.stderr: 3: pthread_cond_wait() 2026-02-26T02:43:25.077 INFO:tasks.ceph.osd.2.trial043.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7faecd0d56c0] 2026-02-26T02:43:25.078 INFO:tasks.ceph.osd.2.trial043.stderr: 5: (AsyncMessenger::wait()+0x73) [0x55710c32d033] 2026-02-26T02:43:25.078 INFO:tasks.ceph.osd.2.trial043.stderr: 6: main() 2026-02-26T02:43:25.078 INFO:tasks.ceph.osd.2.trial043.stderr: 7: /lib64/libc.so.6(+0x2a610) [0x7faeccc2a610] 2026-02-26T02:43:25.078 INFO:tasks.ceph.osd.2.trial043.stderr: 8: __libc_start_main() 2026-02-26T02:43:25.078 INFO:tasks.ceph.osd.2.trial043.stderr: 9: _start() 2026-02-26T02:43:25.078 INFO:tasks.ceph.osd.0.trial043.stderr: ceph version 19.2.3-687-g8d724284 (8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5) squid (stable) 2026-02-26T02:43:25.078 INFO:tasks.ceph.osd.0.trial043.stderr: 1: /lib64/libc.so.6(+0x3fc30) [0x7ff7a303fc30] 2026-02-26T02:43:25.078 INFO:tasks.ceph.osd.0.trial043.stderr: 2: /lib64/libc.so.6(+0x8838a) [0x7ff7a308838a] 2026-02-26T02:43:25.079 INFO:tasks.ceph.osd.0.trial043.stderr: 3: pthread_cond_wait() 2026-02-26T02:43:25.079 INFO:tasks.ceph.osd.0.trial043.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7ff7a34d56c0] 2026-02-26T02:43:25.079 INFO:tasks.ceph.osd.0.trial043.stderr: 5: (AsyncMessenger::wait()+0x73) [0x562df0401033] 2026-02-26T02:43:25.079 INFO:tasks.ceph.osd.0.trial043.stderr: 6: main() 2026-02-26T02:43:25.079 INFO:tasks.ceph.osd.0.trial043.stderr: 7: /lib64/libc.so.6(+0x2a610) [0x7ff7a302a610] 2026-02-26T02:43:25.079 INFO:tasks.ceph.osd.0.trial043.stderr: 8: __libc_start_main() 2026-02-26T02:43:25.079 INFO:tasks.ceph.osd.0.trial043.stderr: 9: _start() 2026-02-26T02:43:25.079 INFO:tasks.ceph.osd.1.trial043.stderr: ceph version 19.2.3-687-g8d724284 (8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5) squid (stable) 2026-02-26T02:43:25.079 INFO:tasks.ceph.osd.1.trial043.stderr: 1: /lib64/libc.so.6(+0x3fc30) [0x7f7ae903fc30] 2026-02-26T02:43:25.080 INFO:tasks.ceph.osd.1.trial043.stderr: 2: /lib64/libc.so.6(+0x8838a) [0x7f7ae908838a] 2026-02-26T02:43:25.080 INFO:tasks.ceph.osd.1.trial043.stderr: 3: pthread_cond_wait() 2026-02-26T02:43:25.080 INFO:tasks.ceph.osd.1.trial043.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f7ae94d56c0] 2026-02-26T02:43:25.080 INFO:tasks.ceph.osd.1.trial043.stderr: 5: (AsyncMessenger::wait()+0x73) [0x558ea0a4d033] 2026-02-26T02:43:25.080 INFO:tasks.ceph.osd.1.trial043.stderr: 6: main() 2026-02-26T02:43:25.080 INFO:tasks.ceph.osd.1.trial043.stderr: 7: /lib64/libc.so.6(+0x2a610) [0x7f7ae902a610] 2026-02-26T02:43:25.080 INFO:tasks.ceph.osd.1.trial043.stderr: 8: __libc_start_main() 2026-02-26T02:43:25.080 INFO:tasks.ceph.osd.1.trial043.stderr: 9: _start() 2026-02-26T02:43:25.081 INFO:tasks.ceph.osd.2.trial043.stderr:2026-02-26T02:43:25.073+0000 7faecce11780 -1 *** Caught signal (Aborted) ** 2026-02-26T02:43:25.081 INFO:tasks.ceph.osd.2.trial043.stderr: in thread 7faecce11780 thread_name:ceph-osd 2026-02-26T02:43:25.081 INFO:tasks.ceph.osd.2.trial043.stderr: 2026-02-26T02:43:25.081 INFO:tasks.ceph.osd.2.trial043.stderr: ceph version 19.2.3-687-g8d724284 (8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5) squid (stable) 2026-02-26T02:43:25.081 INFO:tasks.ceph.osd.2.trial043.stderr: 1: /lib64/libc.so.6(+0x3fc30) [0x7faeccc3fc30] 2026-02-26T02:43:25.081 INFO:tasks.ceph.osd.2.trial043.stderr: 2: /lib64/libc.so.6(+0x8838a) [0x7faeccc8838a] 2026-02-26T02:43:25.081 INFO:tasks.ceph.osd.2.trial043.stderr: 3: pthread_cond_wait() 2026-02-26T02:43:25.081 INFO:tasks.ceph.osd.2.trial043.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7faecd0d56c0] 2026-02-26T02:43:25.081 INFO:tasks.ceph.osd.2.trial043.stderr: 5: (AsyncMessenger::wait()+0x73) [0x55710c32d033] 2026-02-26T02:43:25.082 INFO:tasks.ceph.osd.2.trial043.stderr: 6: main() 2026-02-26T02:43:25.082 INFO:tasks.ceph.osd.2.trial043.stderr: 7: /lib64/libc.so.6(+0x2a610) [0x7faeccc2a610] 2026-02-26T02:43:25.082 INFO:tasks.ceph.osd.2.trial043.stderr: 8: __libc_start_main() 2026-02-26T02:43:25.082 INFO:tasks.ceph.osd.2.trial043.stderr: 9: _start() 2026-02-26T02:43:25.082 INFO:tasks.ceph.osd.2.trial043.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2026-02-26T02:43:25.082 INFO:tasks.ceph.osd.2.trial043.stderr: 2026-02-26T02:43:25.082 INFO:tasks.ceph.osd.0.trial043.stderr:2026-02-26T02:43:25.073+0000 7ff7a2b34780 -1 *** Caught signal (Aborted) ** 2026-02-26T02:43:25.083 INFO:tasks.ceph.osd.0.trial043.stderr: in thread 7ff7a2b34780 thread_name:ceph-osd 2026-02-26T02:43:25.083 INFO:tasks.ceph.osd.0.trial043.stderr: 2026-02-26T02:43:25.083 INFO:tasks.ceph.osd.0.trial043.stderr: ceph version 19.2.3-687-g8d724284 (8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5) squid (stable) 2026-02-26T02:43:25.083 INFO:tasks.ceph.osd.0.trial043.stderr: 1: /lib64/libc.so.6(+0x3fc30) [0x7ff7a303fc30] 2026-02-26T02:43:25.083 INFO:tasks.ceph.osd.0.trial043.stderr: 2: /lib64/libc.so.6(+0x8838a) [0x7ff7a308838a] 2026-02-26T02:43:25.083 INFO:tasks.ceph.osd.0.trial043.stderr: 3: pthread_cond_wait() 2026-02-26T02:43:25.083 INFO:tasks.ceph.osd.0.trial043.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7ff7a34d56c0] 2026-02-26T02:43:25.083 INFO:tasks.ceph.osd.0.trial043.stderr: 5: (AsyncMessenger::wait()+0x73) [0x562df0401033] 2026-02-26T02:43:25.084 INFO:tasks.ceph.osd.0.trial043.stderr: 6: main() 2026-02-26T02:43:25.084 INFO:tasks.ceph.osd.0.trial043.stderr: 7: /lib64/libc.so.6(+0x2a610) [0x7ff7a302a610] 2026-02-26T02:43:25.084 INFO:tasks.ceph.osd.0.trial043.stderr: 8: __libc_start_main() 2026-02-26T02:43:25.084 INFO:tasks.ceph.osd.0.trial043.stderr: 9: _start() 2026-02-26T02:43:25.084 INFO:tasks.ceph.osd.0.trial043.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2026-02-26T02:43:25.084 INFO:tasks.ceph.osd.0.trial043.stderr: 2026-02-26T02:43:25.084 INFO:tasks.ceph.osd.1.trial043.stderr:2026-02-26T02:43:25.073+0000 7f7ae8bf5780 -1 *** Caught signal (Aborted) ** 2026-02-26T02:43:25.085 INFO:tasks.ceph.osd.1.trial043.stderr: in thread 7f7ae8bf5780 thread_name:ceph-osd 2026-02-26T02:43:25.085 INFO:tasks.ceph.osd.1.trial043.stderr: 2026-02-26T02:43:25.085 INFO:tasks.ceph.osd.1.trial043.stderr: ceph version 19.2.3-687-g8d724284 (8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5) squid (stable) 2026-02-26T02:43:25.085 INFO:tasks.ceph.osd.1.trial043.stderr: 1: /lib64/libc.so.6(+0x3fc30) [0x7f7ae903fc30] 2026-02-26T02:43:25.085 INFO:tasks.ceph.osd.1.trial043.stderr: 2: /lib64/libc.so.6(+0x8838a) [0x7f7ae908838a] 2026-02-26T02:43:25.085 INFO:tasks.ceph.osd.1.trial043.stderr: 3: pthread_cond_wait() 2026-02-26T02:43:25.085 INFO:tasks.ceph.osd.1.trial043.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f7ae94d56c0] 2026-02-26T02:43:25.085 INFO:tasks.ceph.osd.1.trial043.stderr: 5: (AsyncMessenger::wait()+0x73) [0x558ea0a4d033] 2026-02-26T02:43:25.085 INFO:tasks.ceph.osd.1.trial043.stderr: 6: main() 2026-02-26T02:43:25.085 INFO:tasks.ceph.osd.1.trial043.stderr: 7: /lib64/libc.so.6(+0x2a610) [0x7f7ae902a610] 2026-02-26T02:43:25.086 INFO:tasks.ceph.osd.1.trial043.stderr: 8: __libc_start_main() 2026-02-26T02:43:25.086 INFO:tasks.ceph.osd.1.trial043.stderr: 9: _start() 2026-02-26T02:43:25.086 INFO:tasks.ceph.osd.1.trial043.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2026-02-26T02:43:25.086 INFO:tasks.ceph.osd.1.trial043.stderr: 2026-02-26T02:43:25.086 INFO:tasks.workunit.client.0.trial043.stderr:+ sleep 5 2026-02-26T02:43:25.087 INFO:tasks.ceph.osd.2.trial043.stderr: -9125> 2026-02-26T02:42:47.170+0000 7faecce11780 -1 osd.2 0 log_to_monitors true 2026-02-26T02:43:25.087 INFO:tasks.ceph.osd.2.trial043.stderr: -8773> 2026-02-26T02:42:48.230+0000 7faec8016640 -1 osd.2 0 waiting for initial osdmap 2026-02-26T02:43:25.087 INFO:tasks.ceph.osd.0.trial043.stderr: -9762> 2026-02-26T02:42:48.494+0000 7ff7968d2640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-26T02:43:25.088 INFO:tasks.ceph.osd.2.trial043.stderr: -8150> 2026-02-26T02:42:48.231+0000 7faec4652640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-26T02:43:25.088 INFO:tasks.ceph.osd.2.trial043.stderr: -8014> 2026-02-26T02:42:50.176+0000 7faebf058640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-26T02:43:25.088 INFO:tasks.ceph.osd.2.trial043.stderr: 0> 2026-02-26T02:43:25.073+0000 7faecce11780 -1 *** Caught signal (Aborted) ** 2026-02-26T02:43:25.088 INFO:tasks.ceph.osd.2.trial043.stderr: in thread 7faecce11780 thread_name:ceph-osd 2026-02-26T02:43:25.088 INFO:tasks.ceph.osd.2.trial043.stderr: 2026-02-26T02:43:25.088 INFO:tasks.ceph.osd.2.trial043.stderr: ceph version 19.2.3-687-g8d724284 (8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5) squid (stable) 2026-02-26T02:43:25.089 INFO:tasks.ceph.osd.2.trial043.stderr: 1: /lib64/libc.so.6(+0x3fc30) [0x7faeccc3fc30] 2026-02-26T02:43:25.089 INFO:tasks.ceph.osd.2.trial043.stderr: 2: /lib64/libc.so.6(+0x8838a) [0x7faeccc8838a] 2026-02-26T02:43:25.089 INFO:tasks.ceph.osd.2.trial043.stderr: 3: pthread_cond_wait() 2026-02-26T02:43:25.089 INFO:tasks.ceph.osd.2.trial043.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7faecd0d56c0] 2026-02-26T02:43:25.089 INFO:tasks.ceph.osd.2.trial043.stderr: 5: (AsyncMessenger::wait()+0x73) [0x55710c32d033] 2026-02-26T02:43:25.089 INFO:tasks.ceph.osd.2.trial043.stderr: 6: main() 2026-02-26T02:43:25.089 INFO:tasks.ceph.osd.2.trial043.stderr: 7: /lib64/libc.so.6(+0x2a610) [0x7faeccc2a610] 2026-02-26T02:43:25.089 INFO:tasks.ceph.osd.2.trial043.stderr: 8: __libc_start_main() 2026-02-26T02:43:25.089 INFO:tasks.ceph.osd.2.trial043.stderr: 9: _start() 2026-02-26T02:43:25.089 INFO:tasks.ceph.osd.2.trial043.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2026-02-26T02:43:25.090 INFO:tasks.ceph.osd.2.trial043.stderr: 2026-02-26T02:43:25.090 INFO:tasks.ceph.osd.0.trial043.stderr: 0> 2026-02-26T02:43:25.073+0000 7ff7a2b34780 -1 *** Caught signal (Aborted) ** 2026-02-26T02:43:25.090 INFO:tasks.ceph.osd.0.trial043.stderr: in thread 7ff7a2b34780 thread_name:ceph-osd 2026-02-26T02:43:25.090 INFO:tasks.ceph.osd.0.trial043.stderr: 2026-02-26T02:43:25.090 INFO:tasks.ceph.osd.0.trial043.stderr: ceph version 19.2.3-687-g8d724284 (8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5) squid (stable) 2026-02-26T02:43:25.090 INFO:tasks.ceph.osd.0.trial043.stderr: 1: /lib64/libc.so.6(+0x3fc30) [0x7ff7a303fc30] 2026-02-26T02:43:25.090 INFO:tasks.ceph.osd.0.trial043.stderr: 2: /lib64/libc.so.6(+0x8838a) [0x7ff7a308838a] 2026-02-26T02:43:25.091 INFO:tasks.ceph.osd.0.trial043.stderr: 3: pthread_cond_wait() 2026-02-26T02:43:25.091 INFO:tasks.ceph.osd.0.trial043.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7ff7a34d56c0] 2026-02-26T02:43:25.091 INFO:tasks.ceph.osd.0.trial043.stderr: 5: (AsyncMessenger::wait()+0x73) [0x562df0401033] 2026-02-26T02:43:25.091 INFO:tasks.ceph.osd.0.trial043.stderr: 6: main() 2026-02-26T02:43:25.091 INFO:tasks.ceph.osd.0.trial043.stderr: 7: /lib64/libc.so.6(+0x2a610) [0x7ff7a302a610] 2026-02-26T02:43:25.091 INFO:tasks.ceph.osd.0.trial043.stderr: 8: __libc_start_main() 2026-02-26T02:43:25.091 INFO:tasks.ceph.osd.0.trial043.stderr: 9: _start() 2026-02-26T02:43:25.091 INFO:tasks.ceph.osd.0.trial043.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2026-02-26T02:43:25.091 INFO:tasks.ceph.osd.0.trial043.stderr: 2026-02-26T02:43:25.092 INFO:tasks.ceph.osd.2.trial043.stderr: -9125> 2026-02-26T02:42:47.170+0000 7faecce11780 -1 osd.2 0 log_to_monitors true 2026-02-26T02:43:25.092 INFO:tasks.ceph.osd.2.trial043.stderr: -8773> 2026-02-26T02:42:48.230+0000 7faec8016640 -1 osd.2 0 waiting for initial osdmap 2026-02-26T02:43:25.092 INFO:tasks.ceph.osd.1.trial043.stderr: 0> 2026-02-26T02:43:25.073+0000 7f7ae8bf5780 -1 *** Caught signal (Aborted) ** 2026-02-26T02:43:25.092 INFO:tasks.ceph.osd.1.trial043.stderr: in thread 7f7ae8bf5780 thread_name:ceph-osd 2026-02-26T02:43:25.093 INFO:tasks.ceph.osd.1.trial043.stderr: 2026-02-26T02:43:25.093 INFO:tasks.ceph.osd.1.trial043.stderr: ceph version 19.2.3-687-g8d724284 (8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5) squid (stable) 2026-02-26T02:43:25.093 INFO:tasks.ceph.osd.1.trial043.stderr: 1: /lib64/libc.so.6(+0x3fc30) [0x7f7ae903fc30] 2026-02-26T02:43:25.093 INFO:tasks.ceph.osd.1.trial043.stderr: 2: /lib64/libc.so.6(+0x8838a) [0x7f7ae908838a] 2026-02-26T02:43:25.093 INFO:tasks.ceph.osd.1.trial043.stderr: 3: pthread_cond_wait() 2026-02-26T02:43:25.093 INFO:tasks.ceph.osd.1.trial043.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f7ae94d56c0] 2026-02-26T02:43:25.093 INFO:tasks.ceph.osd.1.trial043.stderr: 5: (AsyncMessenger::wait()+0x73) [0x558ea0a4d033] 2026-02-26T02:43:25.093 INFO:tasks.ceph.osd.1.trial043.stderr: 6: main() 2026-02-26T02:43:25.094 INFO:tasks.ceph.osd.1.trial043.stderr: 7: /lib64/libc.so.6(+0x2a610) [0x7f7ae902a610] 2026-02-26T02:43:25.094 INFO:tasks.ceph.osd.1.trial043.stderr: 8: __libc_start_main() 2026-02-26T02:43:25.094 INFO:tasks.ceph.osd.1.trial043.stderr: 9: _start() 2026-02-26T02:43:25.094 INFO:tasks.ceph.osd.1.trial043.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2026-02-26T02:43:25.094 INFO:tasks.ceph.osd.1.trial043.stderr: 2026-02-26T02:43:25.095 INFO:tasks.ceph.osd.2.trial043.stderr: -8150> 2026-02-26T02:42:48.231+0000 7faec4652640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-26T02:43:25.095 INFO:tasks.ceph.osd.2.trial043.stderr: -8014> 2026-02-26T02:42:50.176+0000 7faebf058640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-26T02:43:25.095 INFO:tasks.ceph.osd.0.trial043.stderr: -9762> 2026-02-26T02:42:48.494+0000 7ff7968d2640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-26T02:43:25.095 INFO:tasks.ceph.osd.2.trial043.stderr: 0> 2026-02-26T02:43:25.073+0000 7faecce11780 -1 *** Caught signal (Aborted) ** 2026-02-26T02:43:25.096 INFO:tasks.ceph.osd.2.trial043.stderr: in thread 7faecce11780 thread_name:ceph-osd 2026-02-26T02:43:25.096 INFO:tasks.ceph.osd.2.trial043.stderr: 2026-02-26T02:43:25.096 INFO:tasks.ceph.osd.2.trial043.stderr: ceph version 19.2.3-687-g8d724284 (8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5) squid (stable) 2026-02-26T02:43:25.096 INFO:tasks.ceph.osd.2.trial043.stderr: 1: /lib64/libc.so.6(+0x3fc30) [0x7faeccc3fc30] 2026-02-26T02:43:25.096 INFO:tasks.ceph.osd.2.trial043.stderr: 2: /lib64/libc.so.6(+0x8838a) [0x7faeccc8838a] 2026-02-26T02:43:25.096 INFO:tasks.ceph.osd.2.trial043.stderr: 3: pthread_cond_wait() 2026-02-26T02:43:25.096 INFO:tasks.ceph.osd.2.trial043.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7faecd0d56c0] 2026-02-26T02:43:25.096 INFO:tasks.ceph.osd.2.trial043.stderr: 5: (AsyncMessenger::wait()+0x73) [0x55710c32d033] 2026-02-26T02:43:25.096 INFO:tasks.ceph.osd.2.trial043.stderr: 6: main() 2026-02-26T02:43:25.096 INFO:tasks.ceph.osd.2.trial043.stderr: 7: /lib64/libc.so.6(+0x2a610) [0x7faeccc2a610] 2026-02-26T02:43:25.097 INFO:tasks.ceph.osd.2.trial043.stderr: 8: __libc_start_main() 2026-02-26T02:43:25.097 INFO:tasks.ceph.osd.2.trial043.stderr: 9: _start() 2026-02-26T02:43:25.097 INFO:tasks.ceph.osd.2.trial043.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2026-02-26T02:43:25.097 INFO:tasks.ceph.osd.2.trial043.stderr: 2026-02-26T02:43:25.097 INFO:tasks.ceph.osd.0.trial043.stderr: 0> 2026-02-26T02:43:25.073+0000 7ff7a2b34780 -1 *** Caught signal (Aborted) ** 2026-02-26T02:43:25.097 INFO:tasks.ceph.osd.0.trial043.stderr: in thread 7ff7a2b34780 thread_name:ceph-osd 2026-02-26T02:43:25.097 INFO:tasks.ceph.osd.0.trial043.stderr: 2026-02-26T02:43:25.098 INFO:tasks.ceph.osd.0.trial043.stderr: ceph version 19.2.3-687-g8d724284 (8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5) squid (stable) 2026-02-26T02:43:25.098 INFO:tasks.ceph.osd.0.trial043.stderr: 1: /lib64/libc.so.6(+0x3fc30) [0x7ff7a303fc30] 2026-02-26T02:43:25.098 INFO:tasks.ceph.osd.0.trial043.stderr: 2: /lib64/libc.so.6(+0x8838a) [0x7ff7a308838a] 2026-02-26T02:43:25.098 INFO:tasks.ceph.osd.0.trial043.stderr: 3: pthread_cond_wait() 2026-02-26T02:43:25.098 INFO:tasks.ceph.osd.0.trial043.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7ff7a34d56c0] 2026-02-26T02:43:25.098 INFO:tasks.ceph.osd.0.trial043.stderr: 5: (AsyncMessenger::wait()+0x73) [0x562df0401033] 2026-02-26T02:43:25.098 INFO:tasks.ceph.osd.0.trial043.stderr: 6: main() 2026-02-26T02:43:25.098 INFO:tasks.ceph.osd.0.trial043.stderr: 7: /lib64/libc.so.6(+0x2a610) [0x7ff7a302a610] 2026-02-26T02:43:25.098 INFO:tasks.ceph.osd.0.trial043.stderr: 8: __libc_start_main() 2026-02-26T02:43:25.098 INFO:tasks.ceph.osd.0.trial043.stderr: 9: _start() 2026-02-26T02:43:25.099 INFO:tasks.ceph.osd.0.trial043.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2026-02-26T02:43:25.099 INFO:tasks.ceph.osd.0.trial043.stderr: 2026-02-26T02:43:25.099 INFO:tasks.ceph.osd.1.trial043.stderr: 0> 2026-02-26T02:43:25.073+0000 7f7ae8bf5780 -1 *** Caught signal (Aborted) ** 2026-02-26T02:43:25.099 INFO:tasks.ceph.osd.1.trial043.stderr: in thread 7f7ae8bf5780 thread_name:ceph-osd 2026-02-26T02:43:25.099 INFO:tasks.ceph.osd.1.trial043.stderr: 2026-02-26T02:43:25.099 INFO:tasks.ceph.osd.1.trial043.stderr: ceph version 19.2.3-687-g8d724284 (8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5) squid (stable) 2026-02-26T02:43:25.099 INFO:tasks.ceph.osd.1.trial043.stderr: 1: /lib64/libc.so.6(+0x3fc30) [0x7f7ae903fc30] 2026-02-26T02:43:25.099 INFO:tasks.ceph.osd.1.trial043.stderr: 2: /lib64/libc.so.6(+0x8838a) [0x7f7ae908838a] 2026-02-26T02:43:25.100 INFO:tasks.ceph.osd.1.trial043.stderr: 3: pthread_cond_wait() 2026-02-26T02:43:25.100 INFO:tasks.ceph.osd.1.trial043.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f7ae94d56c0] 2026-02-26T02:43:25.100 INFO:tasks.ceph.osd.1.trial043.stderr: 5: (AsyncMessenger::wait()+0x73) [0x558ea0a4d033] 2026-02-26T02:43:25.100 INFO:tasks.ceph.osd.1.trial043.stderr: 6: main() 2026-02-26T02:43:25.100 INFO:tasks.ceph.osd.1.trial043.stderr: 7: /lib64/libc.so.6(+0x2a610) [0x7f7ae902a610] 2026-02-26T02:43:25.100 INFO:tasks.ceph.osd.1.trial043.stderr: 8: __libc_start_main() 2026-02-26T02:43:25.100 INFO:tasks.ceph.osd.1.trial043.stderr: 9: _start() 2026-02-26T02:43:25.100 INFO:tasks.ceph.osd.1.trial043.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2026-02-26T02:43:25.100 INFO:tasks.ceph.osd.1.trial043.stderr: 2026-02-26T02:43:25.213 INFO:tasks.ceph.osd.1.trial043.stderr:daemon-helper: command crashed with signal 6 2026-02-26T02:43:25.214 INFO:tasks.ceph.osd.0.trial043.stderr:daemon-helper: command crashed with signal 6 2026-02-26T02:43:25.214 INFO:tasks.ceph.osd.2.trial043.stderr:daemon-helper: command crashed with signal 6 2026-02-26T02:43:28.056 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2026-02-26T02:43:28.056 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2026-02-26T02:43:28.056 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2026-02-26T02:43:30.078 INFO:tasks.workunit.client.0.trial043.stderr:++ find /home/ubuntu/cephtest/archive/coredump -type f 2026-02-26T02:43:30.079 INFO:tasks.workunit.client.0.trial043.stderr:+ for f in $(find $TESTDIR/archive/coredump -type f) 2026-02-26T02:43:30.080 INFO:tasks.workunit.client.0.trial043.stderr:++ echo quit 2026-02-26T02:43:30.080 INFO:tasks.workunit.client.0.trial043.stderr:++ sudo gdb /usr/bin/ceph-osd /home/ubuntu/cephtest/archive/coredump/1772073805.43517.core 2026-02-26T02:43:30.225 INFO:tasks.workunit.client.0.trial043.stderr: 2026-02-26T02:43:30.225 INFO:tasks.workunit.client.0.trial043.stderr:warning: Can't open file /[aio] (deleted) during file-backed mapping note processing 2026-02-26T02:43:30.414 INFO:tasks.workunit.client.0.trial043.stderr:+ gdb_output='GNU gdb (CentOS Stream) 16.3-2.el9 2026-02-26T02:43:30.415 INFO:tasks.workunit.client.0.trial043.stderr:Copyright (C) 2024 Free Software Foundation, Inc. 2026-02-26T02:43:30.415 INFO:tasks.workunit.client.0.trial043.stderr:License GPLv3+: GNU GPL version 3 or later 2026-02-26T02:43:30.415 INFO:tasks.workunit.client.0.trial043.stderr:This is free software: you are free to change and redistribute it. 2026-02-26T02:43:30.415 INFO:tasks.workunit.client.0.trial043.stderr:There is NO WARRANTY, to the extent permitted by law. 2026-02-26T02:43:30.415 INFO:tasks.workunit.client.0.trial043.stderr:Type "show copying" and "show warranty" for details. 2026-02-26T02:43:30.415 INFO:tasks.workunit.client.0.trial043.stderr:This GDB was configured as "x86_64-redhat-linux-gnu". 2026-02-26T02:43:30.415 INFO:tasks.workunit.client.0.trial043.stderr:Type "show configuration" for configuration details. 2026-02-26T02:43:30.415 INFO:tasks.workunit.client.0.trial043.stderr:For bug reporting instructions, please see: 2026-02-26T02:43:30.415 INFO:tasks.workunit.client.0.trial043.stderr:. 2026-02-26T02:43:30.415 INFO:tasks.workunit.client.0.trial043.stderr:Find the GDB manual and other documentation resources online at: 2026-02-26T02:43:30.416 INFO:tasks.workunit.client.0.trial043.stderr: . 2026-02-26T02:43:30.416 INFO:tasks.workunit.client.0.trial043.stderr: 2026-02-26T02:43:30.416 INFO:tasks.workunit.client.0.trial043.stderr:For help, type "help". 2026-02-26T02:43:30.416 INFO:tasks.workunit.client.0.trial043.stderr:Type "apropos word" to search for commands related to "word"... 2026-02-26T02:43:30.416 INFO:tasks.workunit.client.0.trial043.stderr:Reading symbols from /usr/bin/ceph-osd... 2026-02-26T02:43:30.416 INFO:tasks.workunit.client.0.trial043.stderr:Reading symbols from .gnu_debugdata for /usr/bin/ceph-osd... 2026-02-26T02:43:30.416 INFO:tasks.workunit.client.0.trial043.stderr:(No debugging symbols found in .gnu_debugdata for /usr/bin/ceph-osd) 2026-02-26T02:43:30.416 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43517] 2026-02-26T02:43:30.416 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43545] 2026-02-26T02:43:30.416 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43544] 2026-02-26T02:43:30.417 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43551] 2026-02-26T02:43:30.417 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43764] 2026-02-26T02:43:30.417 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43750] 2026-02-26T02:43:30.417 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43749] 2026-02-26T02:43:30.417 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43751] 2026-02-26T02:43:30.417 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43765] 2026-02-26T02:43:30.417 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43826] 2026-02-26T02:43:30.417 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43827] 2026-02-26T02:43:30.417 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43830] 2026-02-26T02:43:30.417 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43542] 2026-02-26T02:43:30.417 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43546] 2026-02-26T02:43:30.418 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43866] 2026-02-26T02:43:30.418 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44981] 2026-02-26T02:43:30.418 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43825] 2026-02-26T02:43:30.418 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44982] 2026-02-26T02:43:30.418 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45534] 2026-02-26T02:43:30.418 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43829] 2026-02-26T02:43:30.418 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43828] 2026-02-26T02:43:30.418 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45745] 2026-02-26T02:43:30.418 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45716] 2026-02-26T02:43:30.418 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45533] 2026-02-26T02:43:30.419 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45724] 2026-02-26T02:43:30.419 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45722] 2026-02-26T02:43:30.419 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43550] 2026-02-26T02:43:30.419 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45758] 2026-02-26T02:43:30.419 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45741] 2026-02-26T02:43:30.419 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45736] 2026-02-26T02:43:30.419 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45720] 2026-02-26T02:43:30.419 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45755] 2026-02-26T02:43:30.419 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45740] 2026-02-26T02:43:30.419 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45749] 2026-02-26T02:43:30.419 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45723] 2026-02-26T02:43:30.419 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45751] 2026-02-26T02:43:30.420 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45743] 2026-02-26T02:43:30.420 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45719] 2026-02-26T02:43:30.420 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45726] 2026-02-26T02:43:30.420 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45728] 2026-02-26T02:43:30.420 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43867] 2026-02-26T02:43:30.420 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43868] 2026-02-26T02:43:30.420 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45717] 2026-02-26T02:43:30.420 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43869] 2026-02-26T02:43:30.420 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45753] 2026-02-26T02:43:30.420 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45721] 2026-02-26T02:43:30.420 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45718] 2026-02-26T02:43:30.421 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45747] 2026-02-26T02:43:30.421 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45725] 2026-02-26T02:43:30.421 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45727] 2026-02-26T02:43:30.421 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45732] 2026-02-26T02:43:30.421 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45729] 2026-02-26T02:43:30.421 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45730] 2026-02-26T02:43:30.421 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45731] 2026-02-26T02:43:30.421 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45715] 2026-02-26T02:43:30.421 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45733] 2026-02-26T02:43:30.421 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45734] 2026-02-26T02:43:30.422 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45735] 2026-02-26T02:43:30.422 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45737] 2026-02-26T02:43:30.422 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45738] 2026-02-26T02:43:30.422 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45739] 2026-02-26T02:43:30.422 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45742] 2026-02-26T02:43:30.422 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45744] 2026-02-26T02:43:30.422 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45746] 2026-02-26T02:43:30.422 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45748] 2026-02-26T02:43:30.422 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45750] 2026-02-26T02:43:30.422 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45752] 2026-02-26T02:43:30.422 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45754] 2026-02-26T02:43:30.422 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45756] 2026-02-26T02:43:30.423 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45757] 2026-02-26T02:43:30.423 INFO:tasks.workunit.client.0.trial043.stderr:[Thread debugging using libthread_db enabled] 2026-02-26T02:43:30.423 INFO:tasks.workunit.client.0.trial043.stderr:Using host libthread_db library "/lib64/libthread_db.so.1". 2026-02-26T02:43:30.423 INFO:tasks.workunit.client.0.trial043.stderr:Core was generated by `/bin/ceph-osd -f --cluster ceph -i 2'\''. 2026-02-26T02:43:30.423 INFO:tasks.workunit.client.0.trial043.stderr:Program terminated with signal SIGABRT, Aborted. 2026-02-26T02:43:30.423 INFO:tasks.workunit.client.0.trial043.stderr:#0 0x00007faeccc8d03c in __pthread_kill_implementation () 2026-02-26T02:43:30.423 INFO:tasks.workunit.client.0.trial043.stderr: from /lib64/libc.so.6 2026-02-26T02:43:30.423 INFO:tasks.workunit.client.0.trial043.stderr:[Current thread is 1 (Thread 0x7faecce11780 (LWP 43517))] 2026-02-26T02:43:30.423 INFO:tasks.workunit.client.0.trial043.stderr:Missing rpms, try: dnf --enablerepo='\''*debug*'\'' install ceph-osd-debuginfo-19.2.3-687.g8d724284.el9.x86_64 gperftools-libs-debuginfo-2.9.1-3.el9.x86_64 libblkid-debuginfo-2.37.4-21.el9.x86_64 fuse-libs-debuginfo-2.9.9-17.el9.x86_64 libaio-debuginfo-0.3.111-13.el9.x86_64 snappy-debuginfo-1.1.8-8.el9.x86_64 lz4-libs-debuginfo-1.9.3-5.el9.x86_64 zlib-debuginfo-1.2.11-41.el9.x86_64 libcurl-debuginfo-7.76.1-40.el9.x86_64 thrift-debuginfo-0.15.0-4.el9.x86_64 libcap-debuginfo-2.48-10.el9.x86_64 glibc-debuginfo-2.34-245.el9.x86_64 openssl-libs-debuginfo-3.5.5-1.el9.x86_64 systemd-libs-debuginfo-252-64.el9.x86_64 libibverbs-debuginfo-61.0-2.el9.x86_64 librdmacm-debuginfo-61.0-2.el9.x86_64 libstdc++-debuginfo-11.5.0-14.el9.x86_64 libgcc-debuginfo-11.5.0-14.el9.x86_64 libunwind-debuginfo-1.6.2-1.el9.x86_64 libnghttp2-debuginfo-1.43.0-6.el9.x86_64 libidn2-debuginfo-2.3.0-7.el9.x86_64 libssh-debuginfo-0.10.4-17.el9.x86_64 libpsl-debuginfo-0.21.1-5.el9.x86_64 krb5-libs-debuginfo-1.21.1-8.el9.x86_64 libcom_err-debuginfo-1.46.5-8.el9.x86_64 openldap-debuginfo-2.6.8-4.el9.x86_64 libbrotli-debuginfo-1.0.9-9.el9.x86_64 libnl3-debuginfo-3.11.0-1.el9.x86_64 libunistring-debuginfo-0.9.10-15.el9.x86_64 keyutils-libs-debuginfo-1.6.3-1.el9.x86_64 libevent-debuginfo-2.1.12-8.el9.x86_64 cyrus-sasl-lib-debuginfo-2.1.27-21.el9.x86_64 libselinux-debuginfo-3.6-3.el9.x86_64 libxcrypt-debuginfo-4.4.18-3.el9.x86_64 pcre2-debuginfo-10.40-6.el9.x86_64 lttng-ust-debuginfo-2.12.0-6.el9.x86_64 numactl-libs-debuginfo-2.0.19-3.el9.x86_64 userspace-rcu-debuginfo-0.12.1-6.el9.x86_64 ceph-base-debuginfo-19.2.3-687.g8d724284.el9.x86_64 qatzip-libs-debuginfo-1.3.1-1.el9.x86_64 qatlib-debuginfo-25.08.0-2.el9.x86_64 liboath-debuginfo-2.6.12-1.el9.x86_64 2026-02-26T02:43:30.423 INFO:tasks.workunit.client.0.trial043.stderr:(gdb) ' 2026-02-26T02:43:30.423 INFO:tasks.workunit.client.0.trial043.stderr:+ expr match 'GNU gdb (CentOS Stream) 16.3-2.el9 2026-02-26T02:43:30.424 INFO:tasks.workunit.client.0.trial043.stderr:Copyright (C) 2024 Free Software Foundation, Inc. 2026-02-26T02:43:30.424 INFO:tasks.workunit.client.0.trial043.stderr:License GPLv3+: GNU GPL version 3 or later 2026-02-26T02:43:30.424 INFO:tasks.workunit.client.0.trial043.stderr:This is free software: you are free to change and redistribute it. 2026-02-26T02:43:30.424 INFO:tasks.workunit.client.0.trial043.stderr:There is NO WARRANTY, to the extent permitted by law. 2026-02-26T02:43:30.424 INFO:tasks.workunit.client.0.trial043.stderr:Type "show copying" and "show warranty" for details. 2026-02-26T02:43:30.424 INFO:tasks.workunit.client.0.trial043.stderr:This GDB was configured as "x86_64-redhat-linux-gnu". 2026-02-26T02:43:30.424 INFO:tasks.workunit.client.0.trial043.stderr:Type "show configuration" for configuration details. 2026-02-26T02:43:30.424 INFO:tasks.workunit.client.0.trial043.stderr:For bug reporting instructions, please see: 2026-02-26T02:43:30.424 INFO:tasks.workunit.client.0.trial043.stderr:. 2026-02-26T02:43:30.424 INFO:tasks.workunit.client.0.trial043.stderr:Find the GDB manual and other documentation resources online at: 2026-02-26T02:43:30.425 INFO:tasks.workunit.client.0.trial043.stderr: . 2026-02-26T02:43:30.425 INFO:tasks.workunit.client.0.trial043.stderr: 2026-02-26T02:43:30.425 INFO:tasks.workunit.client.0.trial043.stderr:For help, type "help". 2026-02-26T02:43:30.425 INFO:tasks.workunit.client.0.trial043.stderr:Type "apropos word" to search for commands related to "word"... 2026-02-26T02:43:30.425 INFO:tasks.workunit.client.0.trial043.stderr:Reading symbols from /usr/bin/ceph-osd... 2026-02-26T02:43:30.425 INFO:tasks.workunit.client.0.trial043.stderr:Reading symbols from .gnu_debugdata for /usr/bin/ceph-osd... 2026-02-26T02:43:30.425 INFO:tasks.workunit.client.0.trial043.stderr:(No debugging symbols found in .gnu_debugdata for /usr/bin/ceph-osd) 2026-02-26T02:43:30.425 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43517] 2026-02-26T02:43:30.425 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43545] 2026-02-26T02:43:30.425 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43544] 2026-02-26T02:43:30.425 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43551] 2026-02-26T02:43:30.426 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43764] 2026-02-26T02:43:30.426 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43750] 2026-02-26T02:43:30.426 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43749] 2026-02-26T02:43:30.426 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43751] 2026-02-26T02:43:30.426 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43765] 2026-02-26T02:43:30.426 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43826] 2026-02-26T02:43:30.426 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43827] 2026-02-26T02:43:30.426 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43830] 2026-02-26T02:43:30.426 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43542] 2026-02-26T02:43:30.426 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43546] 2026-02-26T02:43:30.426 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43866] 2026-02-26T02:43:30.427 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44981] 2026-02-26T02:43:30.427 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43825] 2026-02-26T02:43:30.427 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44982] 2026-02-26T02:43:30.427 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45534] 2026-02-26T02:43:30.427 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43829] 2026-02-26T02:43:30.427 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43828] 2026-02-26T02:43:30.427 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45745] 2026-02-26T02:43:30.427 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45716] 2026-02-26T02:43:30.427 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45533] 2026-02-26T02:43:30.427 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45724] 2026-02-26T02:43:30.427 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45722] 2026-02-26T02:43:30.427 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43550] 2026-02-26T02:43:30.428 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45758] 2026-02-26T02:43:30.428 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45741] 2026-02-26T02:43:30.428 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45736] 2026-02-26T02:43:30.428 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45720] 2026-02-26T02:43:30.428 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45755] 2026-02-26T02:43:30.428 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45740] 2026-02-26T02:43:30.428 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45749] 2026-02-26T02:43:30.428 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45723] 2026-02-26T02:43:30.428 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45751] 2026-02-26T02:43:30.428 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45743] 2026-02-26T02:43:30.428 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45719] 2026-02-26T02:43:30.429 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45726] 2026-02-26T02:43:30.429 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45728] 2026-02-26T02:43:30.429 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43867] 2026-02-26T02:43:30.429 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43868] 2026-02-26T02:43:30.429 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45717] 2026-02-26T02:43:30.429 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43869] 2026-02-26T02:43:30.429 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45753] 2026-02-26T02:43:30.429 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45721] 2026-02-26T02:43:30.429 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45718] 2026-02-26T02:43:30.429 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45747] 2026-02-26T02:43:30.429 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45725] 2026-02-26T02:43:30.430 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45727] 2026-02-26T02:43:30.430 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45732] 2026-02-26T02:43:30.430 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45729] 2026-02-26T02:43:30.430 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45730] 2026-02-26T02:43:30.430 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45731] 2026-02-26T02:43:30.430 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45715] 2026-02-26T02:43:30.430 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45733] 2026-02-26T02:43:30.430 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45734] 2026-02-26T02:43:30.430 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45735] 2026-02-26T02:43:30.430 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45737] 2026-02-26T02:43:30.430 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45738] 2026-02-26T02:43:30.430 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45739] 2026-02-26T02:43:30.431 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45742] 2026-02-26T02:43:30.431 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45744] 2026-02-26T02:43:30.431 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45746] 2026-02-26T02:43:30.431 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45748] 2026-02-26T02:43:30.431 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45750] 2026-02-26T02:43:30.431 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45752] 2026-02-26T02:43:30.431 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45754] 2026-02-26T02:43:30.431 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45756] 2026-02-26T02:43:30.431 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45757] 2026-02-26T02:43:30.431 INFO:tasks.workunit.client.0.trial043.stderr:[Thread debugging using libthread_db enabled] 2026-02-26T02:43:30.431 INFO:tasks.workunit.client.0.trial043.stderr:Using host libthread_db library "/lib64/libthread_db.so.1". 2026-02-26T02:43:30.432 INFO:tasks.workunit.client.0.trial043.stderr:Core was generated by `/bin/ceph-osd -f --cluster ceph -i 2'\''. 2026-02-26T02:43:30.432 INFO:tasks.workunit.client.0.trial043.stderr:Program terminated with signal SIGABRT, Aborted. 2026-02-26T02:43:30.432 INFO:tasks.workunit.client.0.trial043.stderr:#0 0x00007faeccc8d03c in __pthread_kill_implementation () 2026-02-26T02:43:30.432 INFO:tasks.workunit.client.0.trial043.stderr: from /lib64/libc.so.6 2026-02-26T02:43:30.432 INFO:tasks.workunit.client.0.trial043.stderr:[Current thread is 1 (Thread 0x7faecce11780 (LWP 43517))] 2026-02-26T02:43:30.432 INFO:tasks.workunit.client.0.trial043.stderr:Missing rpms, try: dnf --enablerepo='\''*debug*'\'' install ceph-osd-debuginfo-19.2.3-687.g8d724284.el9.x86_64 gperftools-libs-debuginfo-2.9.1-3.el9.x86_64 libblkid-debuginfo-2.37.4-21.el9.x86_64 fuse-libs-debuginfo-2.9.9-17.el9.x86_64 libaio-debuginfo-0.3.111-13.el9.x86_64 snappy-debuginfo-1.1.8-8.el9.x86_64 lz4-libs-debuginfo-1.9.3-5.el9.x86_64 zlib-debuginfo-1.2.11-41.el9.x86_64 libcurl-debuginfo-7.76.1-40.el9.x86_64 thrift-debuginfo-0.15.0-4.el9.x86_64 libcap-debuginfo-2.48-10.el9.x86_64 glibc-debuginfo-2.34-245.el9.x86_64 openssl-libs-debuginfo-3.5.5-1.el9.x86_64 systemd-libs-debuginfo-252-64.el9.x86_64 libibverbs-debuginfo-61.0-2.el9.x86_64 librdmacm-debuginfo-61.0-2.el9.x86_64 libstdc++-debuginfo-11.5.0-14.el9.x86_64 libgcc-debuginfo-11.5.0-14.el9.x86_64 libunwind-debuginfo-1.6.2-1.el9.x86_64 libnghttp2-debuginfo-1.43.0-6.el9.x86_64 libidn2-debuginfo-2.3.0-7.el9.x86_64 libssh-debuginfo-0.10.4-17.el9.x86_64 libpsl-debuginfo-0.21.1-5.el9.x86_64 krb5-libs-debuginfo-1.21.1-8.el9.x86_64 libcom_err-debuginfo-1.46.5-8.el9.x86_64 openldap-debuginfo-2.6.8-4.el9.x86_64 libbrotli-debuginfo-1.0.9-9.el9.x86_64 libnl3-debuginfo-3.11.0-1.el9.x86_64 libunistring-debuginfo-0.9.10-15.el9.x86_64 keyutils-libs-debuginfo-1.6.3-1.el9.x86_64 libevent-debuginfo-2.1.12-8.el9.x86_64 cyrus-sasl-lib-debuginfo-2.1.27-21.el9.x86_64 libselinux-debuginfo-3.6-3.el9.x86_64 libxcrypt-debuginfo-4.4.18-3.el9.x86_64 pcre2-debuginfo-10.40-6.el9.x86_64 lttng-ust-debuginfo-2.12.0-6.el9.x86_64 numactl-libs-debuginfo-2.0.19-3.el9.x86_64 userspace-rcu-debuginfo-0.12.1-6.el9.x86_64 ceph-base-debuginfo-19.2.3-687.g8d724284.el9.x86_64 qatzip-libs-debuginfo-1.3.1-1.el9.x86_64 qatlib-debuginfo-25.08.0-2.el9.x86_64 liboath-debuginfo-2.6.12-1.el9.x86_64 2026-02-26T02:43:30.432 INFO:tasks.workunit.client.0.trial043.stderr:(gdb) ' '.*generated.*ceph-osd.*' 2026-02-26T02:43:30.433 INFO:tasks.workunit.client.0.trial043.stdout:4128 2026-02-26T02:43:30.433 INFO:tasks.workunit.client.0.trial043.stderr:+ expr match 'GNU gdb (CentOS Stream) 16.3-2.el9 2026-02-26T02:43:30.433 INFO:tasks.workunit.client.0.trial043.stderr:Copyright (C) 2024 Free Software Foundation, Inc. 2026-02-26T02:43:30.434 INFO:tasks.workunit.client.0.trial043.stderr:License GPLv3+: GNU GPL version 3 or later 2026-02-26T02:43:30.434 INFO:tasks.workunit.client.0.trial043.stderr:This is free software: you are free to change and redistribute it. 2026-02-26T02:43:30.434 INFO:tasks.workunit.client.0.trial043.stderr:There is NO WARRANTY, to the extent permitted by law. 2026-02-26T02:43:30.434 INFO:tasks.workunit.client.0.trial043.stderr:Type "show copying" and "show warranty" for details. 2026-02-26T02:43:30.434 INFO:tasks.workunit.client.0.trial043.stderr:This GDB was configured as "x86_64-redhat-linux-gnu". 2026-02-26T02:43:30.434 INFO:tasks.workunit.client.0.trial043.stderr:Type "show configuration" for configuration details. 2026-02-26T02:43:30.434 INFO:tasks.workunit.client.0.trial043.stderr:For bug reporting instructions, please see: 2026-02-26T02:43:30.434 INFO:tasks.workunit.client.0.trial043.stderr:. 2026-02-26T02:43:30.434 INFO:tasks.workunit.client.0.trial043.stderr:Find the GDB manual and other documentation resources online at: 2026-02-26T02:43:30.434 INFO:tasks.workunit.client.0.trial043.stderr: . 2026-02-26T02:43:30.434 INFO:tasks.workunit.client.0.trial043.stderr: 2026-02-26T02:43:30.435 INFO:tasks.workunit.client.0.trial043.stderr:For help, type "help". 2026-02-26T02:43:30.435 INFO:tasks.workunit.client.0.trial043.stderr:Type "apropos word" to search for commands related to "word"... 2026-02-26T02:43:30.435 INFO:tasks.workunit.client.0.trial043.stderr:Reading symbols from /usr/bin/ceph-osd... 2026-02-26T02:43:30.435 INFO:tasks.workunit.client.0.trial043.stderr:Reading symbols from .gnu_debugdata for /usr/bin/ceph-osd... 2026-02-26T02:43:30.435 INFO:tasks.workunit.client.0.trial043.stderr:(No debugging symbols found in .gnu_debugdata for /usr/bin/ceph-osd) 2026-02-26T02:43:30.435 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43517] 2026-02-26T02:43:30.435 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43545] 2026-02-26T02:43:30.435 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43544] 2026-02-26T02:43:30.435 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43551] 2026-02-26T02:43:30.435 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43764] 2026-02-26T02:43:30.435 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43750] 2026-02-26T02:43:30.436 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43749] 2026-02-26T02:43:30.436 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43751] 2026-02-26T02:43:30.436 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43765] 2026-02-26T02:43:30.436 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43826] 2026-02-26T02:43:30.436 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43827] 2026-02-26T02:43:30.436 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43830] 2026-02-26T02:43:30.436 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43542] 2026-02-26T02:43:30.436 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43546] 2026-02-26T02:43:30.436 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43866] 2026-02-26T02:43:30.436 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44981] 2026-02-26T02:43:30.436 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43825] 2026-02-26T02:43:30.436 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44982] 2026-02-26T02:43:30.437 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45534] 2026-02-26T02:43:30.437 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43829] 2026-02-26T02:43:30.437 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43828] 2026-02-26T02:43:30.437 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45745] 2026-02-26T02:43:30.437 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45716] 2026-02-26T02:43:30.437 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45533] 2026-02-26T02:43:30.437 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45724] 2026-02-26T02:43:30.437 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45722] 2026-02-26T02:43:30.437 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43550] 2026-02-26T02:43:30.437 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45758] 2026-02-26T02:43:30.437 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45741] 2026-02-26T02:43:30.437 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45736] 2026-02-26T02:43:30.438 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45720] 2026-02-26T02:43:30.438 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45755] 2026-02-26T02:43:30.438 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45740] 2026-02-26T02:43:30.438 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45749] 2026-02-26T02:43:30.438 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45723] 2026-02-26T02:43:30.438 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45751] 2026-02-26T02:43:30.438 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45743] 2026-02-26T02:43:30.438 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45719] 2026-02-26T02:43:30.438 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45726] 2026-02-26T02:43:30.438 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45728] 2026-02-26T02:43:30.438 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43867] 2026-02-26T02:43:30.438 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43868] 2026-02-26T02:43:30.439 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45717] 2026-02-26T02:43:30.439 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43869] 2026-02-26T02:43:30.439 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45753] 2026-02-26T02:43:30.439 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45721] 2026-02-26T02:43:30.439 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45718] 2026-02-26T02:43:30.439 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45747] 2026-02-26T02:43:30.439 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45725] 2026-02-26T02:43:30.439 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45727] 2026-02-26T02:43:30.439 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45732] 2026-02-26T02:43:30.439 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45729] 2026-02-26T02:43:30.439 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45730] 2026-02-26T02:43:30.439 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45731] 2026-02-26T02:43:30.439 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45715] 2026-02-26T02:43:30.440 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45733] 2026-02-26T02:43:30.440 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45734] 2026-02-26T02:43:30.440 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45735] 2026-02-26T02:43:30.440 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45737] 2026-02-26T02:43:30.440 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45738] 2026-02-26T02:43:30.440 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45739] 2026-02-26T02:43:30.440 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45742] 2026-02-26T02:43:30.440 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45744] 2026-02-26T02:43:30.440 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45746] 2026-02-26T02:43:30.440 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45748] 2026-02-26T02:43:30.440 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45750] 2026-02-26T02:43:30.440 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45752] 2026-02-26T02:43:30.440 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45754] 2026-02-26T02:43:30.441 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45756] 2026-02-26T02:43:30.441 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45757] 2026-02-26T02:43:30.441 INFO:tasks.workunit.client.0.trial043.stderr:[Thread debugging using libthread_db enabled] 2026-02-26T02:43:30.441 INFO:tasks.workunit.client.0.trial043.stderr:Using host libthread_db library "/lib64/libthread_db.so.1". 2026-02-26T02:43:30.441 INFO:tasks.workunit.client.0.trial043.stderr:Core was generated by `/bin/ceph-osd -f --cluster ceph -i 2'\''. 2026-02-26T02:43:30.441 INFO:tasks.workunit.client.0.trial043.stderr:Program terminated with signal SIGABRT, Aborted. 2026-02-26T02:43:30.441 INFO:tasks.workunit.client.0.trial043.stderr:#0 0x00007faeccc8d03c in __pthread_kill_implementation () 2026-02-26T02:43:30.441 INFO:tasks.workunit.client.0.trial043.stderr: from /lib64/libc.so.6 2026-02-26T02:43:30.441 INFO:tasks.workunit.client.0.trial043.stderr:[Current thread is 1 (Thread 0x7faecce11780 (LWP 43517))] 2026-02-26T02:43:30.441 INFO:tasks.workunit.client.0.trial043.stderr:Missing rpms, try: dnf --enablerepo='\''*debug*'\'' install ceph-osd-debuginfo-19.2.3-687.g8d724284.el9.x86_64 gperftools-libs-debuginfo-2.9.1-3.el9.x86_64 libblkid-debuginfo-2.37.4-21.el9.x86_64 fuse-libs-debuginfo-2.9.9-17.el9.x86_64 libaio-debuginfo-0.3.111-13.el9.x86_64 snappy-debuginfo-1.1.8-8.el9.x86_64 lz4-libs-debuginfo-1.9.3-5.el9.x86_64 zlib-debuginfo-1.2.11-41.el9.x86_64 libcurl-debuginfo-7.76.1-40.el9.x86_64 thrift-debuginfo-0.15.0-4.el9.x86_64 libcap-debuginfo-2.48-10.el9.x86_64 glibc-debuginfo-2.34-245.el9.x86_64 openssl-libs-debuginfo-3.5.5-1.el9.x86_64 systemd-libs-debuginfo-252-64.el9.x86_64 libibverbs-debuginfo-61.0-2.el9.x86_64 librdmacm-debuginfo-61.0-2.el9.x86_64 libstdc++-debuginfo-11.5.0-14.el9.x86_64 libgcc-debuginfo-11.5.0-14.el9.x86_64 libunwind-debuginfo-1.6.2-1.el9.x86_64 libnghttp2-debuginfo-1.43.0-6.el9.x86_64 libidn2-debuginfo-2.3.0-7.el9.x86_64 libssh-debuginfo-0.10.4-17.el9.x86_64 libpsl-debuginfo-0.21.1-5.el9.x86_64 krb5-libs-debuginfo-1.21.1-8.el9.x86_64 libcom_err-debuginfo-1.46.5-8.el9.x86_64 openldap-debuginfo-2.6.8-4.el9.x86_64 libbrotli-debuginfo-1.0.9-9.el9.x86_64 libnl3-debuginfo-3.11.0-1.el9.x86_64 libunistring-debuginfo-0.9.10-15.el9.x86_64 keyutils-libs-debuginfo-1.6.3-1.el9.x86_64 libevent-debuginfo-2.1.12-8.el9.x86_64 cyrus-sasl-lib-debuginfo-2.1.27-21.el9.x86_64 libselinux-debuginfo-3.6-3.el9.x86_64 libxcrypt-debuginfo-4.4.18-3.el9.x86_64 pcre2-debuginfo-10.40-6.el9.x86_64 lttng-ust-debuginfo-2.12.0-6.el9.x86_64 numactl-libs-debuginfo-2.0.19-3.el9.x86_64 userspace-rcu-debuginfo-0.12.1-6.el9.x86_64 ceph-base-debuginfo-19.2.3-687.g8d724284.el9.x86_64 qatzip-libs-debuginfo-1.3.1-1.el9.x86_64 qatlib-debuginfo-25.08.0-2.el9.x86_64 liboath-debuginfo-2.6.12-1.el9.x86_64 2026-02-26T02:43:30.441 INFO:tasks.workunit.client.0.trial043.stderr:(gdb) ' '.*terminated.*signal 6.*' 2026-02-26T02:43:30.442 INFO:tasks.workunit.client.0.trial043.stdout:0 2026-02-26T02:43:30.442 INFO:tasks.workunit.client.0.trial043.stderr:+ expr match 'GNU gdb (CentOS Stream) 16.3-2.el9 2026-02-26T02:43:30.442 INFO:tasks.workunit.client.0.trial043.stderr:Copyright (C) 2024 Free Software Foundation, Inc. 2026-02-26T02:43:30.442 INFO:tasks.workunit.client.0.trial043.stderr:License GPLv3+: GNU GPL version 3 or later 2026-02-26T02:43:30.442 INFO:tasks.workunit.client.0.trial043.stderr:This is free software: you are free to change and redistribute it. 2026-02-26T02:43:30.443 INFO:tasks.workunit.client.0.trial043.stderr:There is NO WARRANTY, to the extent permitted by law. 2026-02-26T02:43:30.443 INFO:tasks.workunit.client.0.trial043.stderr:Type "show copying" and "show warranty" for details. 2026-02-26T02:43:30.443 INFO:tasks.workunit.client.0.trial043.stderr:This GDB was configured as "x86_64-redhat-linux-gnu". 2026-02-26T02:43:30.443 INFO:tasks.workunit.client.0.trial043.stderr:Type "show configuration" for configuration details. 2026-02-26T02:43:30.443 INFO:tasks.workunit.client.0.trial043.stderr:For bug reporting instructions, please see: 2026-02-26T02:43:30.443 INFO:tasks.workunit.client.0.trial043.stderr:. 2026-02-26T02:43:30.443 INFO:tasks.workunit.client.0.trial043.stderr:Find the GDB manual and other documentation resources online at: 2026-02-26T02:43:30.443 INFO:tasks.workunit.client.0.trial043.stderr: . 2026-02-26T02:43:30.443 INFO:tasks.workunit.client.0.trial043.stderr: 2026-02-26T02:43:30.443 INFO:tasks.workunit.client.0.trial043.stderr:For help, type "help". 2026-02-26T02:43:30.443 INFO:tasks.workunit.client.0.trial043.stderr:Type "apropos word" to search for commands related to "word"... 2026-02-26T02:43:30.444 INFO:tasks.workunit.client.0.trial043.stderr:Reading symbols from /usr/bin/ceph-osd... 2026-02-26T02:43:30.444 INFO:tasks.workunit.client.0.trial043.stderr:Reading symbols from .gnu_debugdata for /usr/bin/ceph-osd... 2026-02-26T02:43:30.444 INFO:tasks.workunit.client.0.trial043.stderr:(No debugging symbols found in .gnu_debugdata for /usr/bin/ceph-osd) 2026-02-26T02:43:30.444 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43517] 2026-02-26T02:43:30.444 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43545] 2026-02-26T02:43:30.444 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43544] 2026-02-26T02:43:30.444 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43551] 2026-02-26T02:43:30.444 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43764] 2026-02-26T02:43:30.444 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43750] 2026-02-26T02:43:30.444 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43749] 2026-02-26T02:43:30.445 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43751] 2026-02-26T02:43:30.445 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43765] 2026-02-26T02:43:30.445 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43826] 2026-02-26T02:43:30.445 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43827] 2026-02-26T02:43:30.445 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43830] 2026-02-26T02:43:30.445 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43542] 2026-02-26T02:43:30.445 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43546] 2026-02-26T02:43:30.445 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43866] 2026-02-26T02:43:30.445 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44981] 2026-02-26T02:43:30.445 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43825] 2026-02-26T02:43:30.445 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44982] 2026-02-26T02:43:30.445 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45534] 2026-02-26T02:43:30.446 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43829] 2026-02-26T02:43:30.446 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43828] 2026-02-26T02:43:30.446 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45745] 2026-02-26T02:43:30.446 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45716] 2026-02-26T02:43:30.446 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45533] 2026-02-26T02:43:30.446 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45724] 2026-02-26T02:43:30.446 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45722] 2026-02-26T02:43:30.446 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43550] 2026-02-26T02:43:30.446 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45758] 2026-02-26T02:43:30.446 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45741] 2026-02-26T02:43:30.447 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45736] 2026-02-26T02:43:30.447 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45720] 2026-02-26T02:43:30.447 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45755] 2026-02-26T02:43:30.447 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45740] 2026-02-26T02:43:30.447 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45749] 2026-02-26T02:43:30.447 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45723] 2026-02-26T02:43:30.447 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45751] 2026-02-26T02:43:30.447 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45743] 2026-02-26T02:43:30.447 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45719] 2026-02-26T02:43:30.447 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45726] 2026-02-26T02:43:30.447 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45728] 2026-02-26T02:43:30.447 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43867] 2026-02-26T02:43:30.448 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43868] 2026-02-26T02:43:30.448 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45717] 2026-02-26T02:43:30.448 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43869] 2026-02-26T02:43:30.448 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45753] 2026-02-26T02:43:30.448 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45721] 2026-02-26T02:43:30.448 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45718] 2026-02-26T02:43:30.448 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45747] 2026-02-26T02:43:30.448 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45725] 2026-02-26T02:43:30.448 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45727] 2026-02-26T02:43:30.448 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45732] 2026-02-26T02:43:30.448 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45729] 2026-02-26T02:43:30.449 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45730] 2026-02-26T02:43:30.449 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45731] 2026-02-26T02:43:30.449 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45715] 2026-02-26T02:43:30.449 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45733] 2026-02-26T02:43:30.449 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45734] 2026-02-26T02:43:30.449 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45735] 2026-02-26T02:43:30.449 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45737] 2026-02-26T02:43:30.449 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45738] 2026-02-26T02:43:30.449 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45739] 2026-02-26T02:43:30.449 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45742] 2026-02-26T02:43:30.449 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45744] 2026-02-26T02:43:30.449 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45746] 2026-02-26T02:43:30.450 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45748] 2026-02-26T02:43:30.450 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45750] 2026-02-26T02:43:30.450 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45752] 2026-02-26T02:43:30.450 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45754] 2026-02-26T02:43:30.450 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45756] 2026-02-26T02:43:30.450 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45757] 2026-02-26T02:43:30.450 INFO:tasks.workunit.client.0.trial043.stderr:[Thread debugging using libthread_db enabled] 2026-02-26T02:43:30.450 INFO:tasks.workunit.client.0.trial043.stderr:Using host libthread_db library "/lib64/libthread_db.so.1". 2026-02-26T02:43:30.450 INFO:tasks.workunit.client.0.trial043.stderr:Core was generated by `/bin/ceph-osd -f --cluster ceph -i 2'\''. 2026-02-26T02:43:30.450 INFO:tasks.workunit.client.0.trial043.stderr:Program terminated with signal SIGABRT, Aborted. 2026-02-26T02:43:30.450 INFO:tasks.workunit.client.0.trial043.stderr:#0 0x00007faeccc8d03c in __pthread_kill_implementation () 2026-02-26T02:43:30.450 INFO:tasks.workunit.client.0.trial043.stderr: from /lib64/libc.so.6 2026-02-26T02:43:30.450 INFO:tasks.workunit.client.0.trial043.stderr:[Current thread is 1 (Thread 0x7faecce11780 (LWP 43517))] 2026-02-26T02:43:30.451 INFO:tasks.workunit.client.0.trial043.stderr:Missing rpms, try: dnf --enablerepo='\''*debug*'\'' install ceph-osd-debuginfo-19.2.3-687.g8d724284.el9.x86_64 gperftools-libs-debuginfo-2.9.1-3.el9.x86_64 libblkid-debuginfo-2.37.4-21.el9.x86_64 fuse-libs-debuginfo-2.9.9-17.el9.x86_64 libaio-debuginfo-0.3.111-13.el9.x86_64 snappy-debuginfo-1.1.8-8.el9.x86_64 lz4-libs-debuginfo-1.9.3-5.el9.x86_64 zlib-debuginfo-1.2.11-41.el9.x86_64 libcurl-debuginfo-7.76.1-40.el9.x86_64 thrift-debuginfo-0.15.0-4.el9.x86_64 libcap-debuginfo-2.48-10.el9.x86_64 glibc-debuginfo-2.34-245.el9.x86_64 openssl-libs-debuginfo-3.5.5-1.el9.x86_64 systemd-libs-debuginfo-252-64.el9.x86_64 libibverbs-debuginfo-61.0-2.el9.x86_64 librdmacm-debuginfo-61.0-2.el9.x86_64 libstdc++-debuginfo-11.5.0-14.el9.x86_64 libgcc-debuginfo-11.5.0-14.el9.x86_64 libunwind-debuginfo-1.6.2-1.el9.x86_64 libnghttp2-debuginfo-1.43.0-6.el9.x86_64 libidn2-debuginfo-2.3.0-7.el9.x86_64 libssh-debuginfo-0.10.4-17.el9.x86_64 libpsl-debuginfo-0.21.1-5.el9.x86_64 krb5-libs-debuginfo-1.21.1-8.el9.x86_64 libcom_err-debuginfo-1.46.5-8.el9.x86_64 openldap-debuginfo-2.6.8-4.el9.x86_64 libbrotli-debuginfo-1.0.9-9.el9.x86_64 libnl3-debuginfo-3.11.0-1.el9.x86_64 libunistring-debuginfo-0.9.10-15.el9.x86_64 keyutils-libs-debuginfo-1.6.3-1.el9.x86_64 libevent-debuginfo-2.1.12-8.el9.x86_64 cyrus-sasl-lib-debuginfo-2.1.27-21.el9.x86_64 libselinux-debuginfo-3.6-3.el9.x86_64 libxcrypt-debuginfo-4.4.18-3.el9.x86_64 pcre2-debuginfo-10.40-6.el9.x86_64 lttng-ust-debuginfo-2.12.0-6.el9.x86_64 numactl-libs-debuginfo-2.0.19-3.el9.x86_64 userspace-rcu-debuginfo-0.12.1-6.el9.x86_64 ceph-base-debuginfo-19.2.3-687.g8d724284.el9.x86_64 qatzip-libs-debuginfo-1.3.1-1.el9.x86_64 qatlib-debuginfo-25.08.0-2.el9.x86_64 liboath-debuginfo-2.6.12-1.el9.x86_64 2026-02-26T02:43:30.451 INFO:tasks.workunit.client.0.trial043.stderr:(gdb) ' '.*terminated.*signal SIGABRT.*' 2026-02-26T02:43:30.451 INFO:tasks.workunit.client.0.trial043.stdout:4128 2026-02-26T02:43:30.452 INFO:tasks.workunit.client.0.trial043.stderr:+ sudo rm /home/ubuntu/cephtest/archive/coredump/1772073805.43517.core 2026-02-26T02:43:30.452 INFO:tasks.workunit.client.0.trial043.stderr:+ for f in $(find $TESTDIR/archive/coredump -type f) 2026-02-26T02:43:30.452 INFO:tasks.workunit.client.0.trial043.stderr:++ echo quit 2026-02-26T02:43:30.452 INFO:tasks.workunit.client.0.trial043.stderr:++ sudo gdb /usr/bin/ceph-osd /home/ubuntu/cephtest/archive/coredump/1772073805.43516.core 2026-02-26T02:43:30.530 INFO:tasks.workunit.client.0.trial043.stderr: 2026-02-26T02:43:30.530 INFO:tasks.workunit.client.0.trial043.stderr:warning: Can't open file /[aio] (deleted) during file-backed mapping note processing 2026-02-26T02:43:30.707 INFO:tasks.workunit.client.0.trial043.stderr:+ gdb_output='GNU gdb (CentOS Stream) 16.3-2.el9 2026-02-26T02:43:30.707 INFO:tasks.workunit.client.0.trial043.stderr:Copyright (C) 2024 Free Software Foundation, Inc. 2026-02-26T02:43:30.707 INFO:tasks.workunit.client.0.trial043.stderr:License GPLv3+: GNU GPL version 3 or later 2026-02-26T02:43:30.707 INFO:tasks.workunit.client.0.trial043.stderr:This is free software: you are free to change and redistribute it. 2026-02-26T02:43:30.707 INFO:tasks.workunit.client.0.trial043.stderr:There is NO WARRANTY, to the extent permitted by law. 2026-02-26T02:43:30.707 INFO:tasks.workunit.client.0.trial043.stderr:Type "show copying" and "show warranty" for details. 2026-02-26T02:43:30.707 INFO:tasks.workunit.client.0.trial043.stderr:This GDB was configured as "x86_64-redhat-linux-gnu". 2026-02-26T02:43:30.708 INFO:tasks.workunit.client.0.trial043.stderr:Type "show configuration" for configuration details. 2026-02-26T02:43:30.708 INFO:tasks.workunit.client.0.trial043.stderr:For bug reporting instructions, please see: 2026-02-26T02:43:30.708 INFO:tasks.workunit.client.0.trial043.stderr:. 2026-02-26T02:43:30.708 INFO:tasks.workunit.client.0.trial043.stderr:Find the GDB manual and other documentation resources online at: 2026-02-26T02:43:30.708 INFO:tasks.workunit.client.0.trial043.stderr: . 2026-02-26T02:43:30.708 INFO:tasks.workunit.client.0.trial043.stderr: 2026-02-26T02:43:30.708 INFO:tasks.workunit.client.0.trial043.stderr:For help, type "help". 2026-02-26T02:43:30.708 INFO:tasks.workunit.client.0.trial043.stderr:Type "apropos word" to search for commands related to "word"... 2026-02-26T02:43:30.708 INFO:tasks.workunit.client.0.trial043.stderr:Reading symbols from /usr/bin/ceph-osd... 2026-02-26T02:43:30.708 INFO:tasks.workunit.client.0.trial043.stderr:Reading symbols from .gnu_debugdata for /usr/bin/ceph-osd... 2026-02-26T02:43:30.708 INFO:tasks.workunit.client.0.trial043.stderr:(No debugging symbols found in .gnu_debugdata for /usr/bin/ceph-osd) 2026-02-26T02:43:30.709 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43516] 2026-02-26T02:43:30.709 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43526] 2026-02-26T02:43:30.709 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43530] 2026-02-26T02:43:30.709 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43538] 2026-02-26T02:43:30.709 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43539] 2026-02-26T02:43:30.709 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43755] 2026-02-26T02:43:30.709 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43528] 2026-02-26T02:43:30.709 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43757] 2026-02-26T02:43:30.709 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43842] 2026-02-26T02:43:30.709 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43843] 2026-02-26T02:43:30.709 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43771] 2026-02-26T02:43:30.709 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43844] 2026-02-26T02:43:30.709 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43529] 2026-02-26T02:43:30.709 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43845] 2026-02-26T02:43:30.710 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43846] 2026-02-26T02:43:30.710 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43847] 2026-02-26T02:43:30.710 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44242] 2026-02-26T02:43:30.710 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43754] 2026-02-26T02:43:30.710 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44245] 2026-02-26T02:43:30.710 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45985] 2026-02-26T02:43:30.710 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44243] 2026-02-26T02:43:30.710 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44244] 2026-02-26T02:43:30.710 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46192] 2026-02-26T02:43:30.710 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46168] 2026-02-26T02:43:30.710 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46202] 2026-02-26T02:43:30.710 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43769] 2026-02-26T02:43:30.710 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46170] 2026-02-26T02:43:30.711 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46196] 2026-02-26T02:43:30.711 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45349] 2026-02-26T02:43:30.711 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46205] 2026-02-26T02:43:30.711 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46200] 2026-02-26T02:43:30.711 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46176] 2026-02-26T02:43:30.711 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46201] 2026-02-26T02:43:30.711 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46172] 2026-02-26T02:43:30.711 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46167] 2026-02-26T02:43:30.711 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46197] 2026-02-26T02:43:30.711 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46195] 2026-02-26T02:43:30.711 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46173] 2026-02-26T02:43:30.711 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46178] 2026-02-26T02:43:30.711 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46175] 2026-02-26T02:43:30.711 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46177] 2026-02-26T02:43:30.712 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46179] 2026-02-26T02:43:30.712 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46180] 2026-02-26T02:43:30.712 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46181] 2026-02-26T02:43:30.712 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46182] 2026-02-26T02:43:30.712 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46204] 2026-02-26T02:43:30.712 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46208] 2026-02-26T02:43:30.712 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45986] 2026-02-26T02:43:30.712 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45350] 2026-02-26T02:43:30.712 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46171] 2026-02-26T02:43:30.712 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46210] 2026-02-26T02:43:30.712 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46209] 2026-02-26T02:43:30.712 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46174] 2026-02-26T02:43:30.712 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46194] 2026-02-26T02:43:30.713 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46188] 2026-02-26T02:43:30.713 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46183] 2026-02-26T02:43:30.713 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46184] 2026-02-26T02:43:30.713 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46185] 2026-02-26T02:43:30.713 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46186] 2026-02-26T02:43:30.713 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46187] 2026-02-26T02:43:30.713 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46169] 2026-02-26T02:43:30.713 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46206] 2026-02-26T02:43:30.713 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46199] 2026-02-26T02:43:30.713 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46189] 2026-02-26T02:43:30.713 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46190] 2026-02-26T02:43:30.713 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46191] 2026-02-26T02:43:30.713 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46193] 2026-02-26T02:43:30.714 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46198] 2026-02-26T02:43:30.714 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46203] 2026-02-26T02:43:30.714 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46207] 2026-02-26T02:43:30.714 INFO:tasks.workunit.client.0.trial043.stderr:[Thread debugging using libthread_db enabled] 2026-02-26T02:43:30.714 INFO:tasks.workunit.client.0.trial043.stderr:Using host libthread_db library "/lib64/libthread_db.so.1". 2026-02-26T02:43:30.714 INFO:tasks.workunit.client.0.trial043.stderr:Core was generated by `/bin/ceph-osd -f --cluster ceph -i 1'\''. 2026-02-26T02:43:30.714 INFO:tasks.workunit.client.0.trial043.stderr:Program terminated with signal SIGABRT, Aborted. 2026-02-26T02:43:30.714 INFO:tasks.workunit.client.0.trial043.stderr:#0 0x00007f7ae908d03c in __pthread_kill_implementation () 2026-02-26T02:43:30.714 INFO:tasks.workunit.client.0.trial043.stderr: from /lib64/libc.so.6 2026-02-26T02:43:30.714 INFO:tasks.workunit.client.0.trial043.stderr:[Current thread is 1 (Thread 0x7f7ae8bf5780 (LWP 43516))] 2026-02-26T02:43:30.715 INFO:tasks.workunit.client.0.trial043.stderr:Missing rpms, try: dnf --enablerepo='\''*debug*'\'' install ceph-osd-debuginfo-19.2.3-687.g8d724284.el9.x86_64 gperftools-libs-debuginfo-2.9.1-3.el9.x86_64 libblkid-debuginfo-2.37.4-21.el9.x86_64 fuse-libs-debuginfo-2.9.9-17.el9.x86_64 libaio-debuginfo-0.3.111-13.el9.x86_64 snappy-debuginfo-1.1.8-8.el9.x86_64 lz4-libs-debuginfo-1.9.3-5.el9.x86_64 zlib-debuginfo-1.2.11-41.el9.x86_64 libcurl-debuginfo-7.76.1-40.el9.x86_64 thrift-debuginfo-0.15.0-4.el9.x86_64 libcap-debuginfo-2.48-10.el9.x86_64 glibc-debuginfo-2.34-245.el9.x86_64 openssl-libs-debuginfo-3.5.5-1.el9.x86_64 systemd-libs-debuginfo-252-64.el9.x86_64 libibverbs-debuginfo-61.0-2.el9.x86_64 librdmacm-debuginfo-61.0-2.el9.x86_64 libstdc++-debuginfo-11.5.0-14.el9.x86_64 libgcc-debuginfo-11.5.0-14.el9.x86_64 libunwind-debuginfo-1.6.2-1.el9.x86_64 libnghttp2-debuginfo-1.43.0-6.el9.x86_64 libidn2-debuginfo-2.3.0-7.el9.x86_64 libssh-debuginfo-0.10.4-17.el9.x86_64 libpsl-debuginfo-0.21.1-5.el9.x86_64 krb5-libs-debuginfo-1.21.1-8.el9.x86_64 libcom_err-debuginfo-1.46.5-8.el9.x86_64 openldap-debuginfo-2.6.8-4.el9.x86_64 libbrotli-debuginfo-1.0.9-9.el9.x86_64 libnl3-debuginfo-3.11.0-1.el9.x86_64 libunistring-debuginfo-0.9.10-15.el9.x86_64 keyutils-libs-debuginfo-1.6.3-1.el9.x86_64 libevent-debuginfo-2.1.12-8.el9.x86_64 cyrus-sasl-lib-debuginfo-2.1.27-21.el9.x86_64 libselinux-debuginfo-3.6-3.el9.x86_64 libxcrypt-debuginfo-4.4.18-3.el9.x86_64 pcre2-debuginfo-10.40-6.el9.x86_64 lttng-ust-debuginfo-2.12.0-6.el9.x86_64 numactl-libs-debuginfo-2.0.19-3.el9.x86_64 userspace-rcu-debuginfo-0.12.1-6.el9.x86_64 ceph-base-debuginfo-19.2.3-687.g8d724284.el9.x86_64 qatzip-libs-debuginfo-1.3.1-1.el9.x86_64 qatlib-debuginfo-25.08.0-2.el9.x86_64 liboath-debuginfo-2.6.12-1.el9.x86_64 2026-02-26T02:43:30.715 INFO:tasks.workunit.client.0.trial043.stderr:(gdb) ' 2026-02-26T02:43:30.715 INFO:tasks.workunit.client.0.trial043.stderr:+ expr match 'GNU gdb (CentOS Stream) 16.3-2.el9 2026-02-26T02:43:30.715 INFO:tasks.workunit.client.0.trial043.stderr:Copyright (C) 2024 Free Software Foundation, Inc. 2026-02-26T02:43:30.715 INFO:tasks.workunit.client.0.trial043.stderr:License GPLv3+: GNU GPL version 3 or later 2026-02-26T02:43:30.715 INFO:tasks.workunit.client.0.trial043.stderr:This is free software: you are free to change and redistribute it. 2026-02-26T02:43:30.715 INFO:tasks.workunit.client.0.trial043.stderr:There is NO WARRANTY, to the extent permitted by law. 2026-02-26T02:43:30.715 INFO:tasks.workunit.client.0.trial043.stderr:Type "show copying" and "show warranty" for details. 2026-02-26T02:43:30.715 INFO:tasks.workunit.client.0.trial043.stderr:This GDB was configured as "x86_64-redhat-linux-gnu". 2026-02-26T02:43:30.715 INFO:tasks.workunit.client.0.trial043.stderr:Type "show configuration" for configuration details. 2026-02-26T02:43:30.715 INFO:tasks.workunit.client.0.trial043.stderr:For bug reporting instructions, please see: 2026-02-26T02:43:30.716 INFO:tasks.workunit.client.0.trial043.stderr:. 2026-02-26T02:43:30.716 INFO:tasks.workunit.client.0.trial043.stderr:Find the GDB manual and other documentation resources online at: 2026-02-26T02:43:30.716 INFO:tasks.workunit.client.0.trial043.stderr: . 2026-02-26T02:43:30.716 INFO:tasks.workunit.client.0.trial043.stderr: 2026-02-26T02:43:30.716 INFO:tasks.workunit.client.0.trial043.stderr:For help, type "help". 2026-02-26T02:43:30.716 INFO:tasks.workunit.client.0.trial043.stderr:Type "apropos word" to search for commands related to "word"... 2026-02-26T02:43:30.716 INFO:tasks.workunit.client.0.trial043.stderr:Reading symbols from /usr/bin/ceph-osd... 2026-02-26T02:43:30.716 INFO:tasks.workunit.client.0.trial043.stderr:Reading symbols from .gnu_debugdata for /usr/bin/ceph-osd... 2026-02-26T02:43:30.716 INFO:tasks.workunit.client.0.trial043.stderr:(No debugging symbols found in .gnu_debugdata for /usr/bin/ceph-osd) 2026-02-26T02:43:30.716 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43516] 2026-02-26T02:43:30.716 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43526] 2026-02-26T02:43:30.717 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43530] 2026-02-26T02:43:30.717 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43538] 2026-02-26T02:43:30.717 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43539] 2026-02-26T02:43:30.717 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43755] 2026-02-26T02:43:30.717 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43528] 2026-02-26T02:43:30.717 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43757] 2026-02-26T02:43:30.717 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43842] 2026-02-26T02:43:30.717 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43843] 2026-02-26T02:43:30.717 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43771] 2026-02-26T02:43:30.717 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43844] 2026-02-26T02:43:30.717 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43529] 2026-02-26T02:43:30.717 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43845] 2026-02-26T02:43:30.718 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43846] 2026-02-26T02:43:30.718 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43847] 2026-02-26T02:43:30.718 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44242] 2026-02-26T02:43:30.718 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43754] 2026-02-26T02:43:30.718 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44245] 2026-02-26T02:43:30.718 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45985] 2026-02-26T02:43:30.718 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44243] 2026-02-26T02:43:30.718 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44244] 2026-02-26T02:43:30.718 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46192] 2026-02-26T02:43:30.718 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46168] 2026-02-26T02:43:30.718 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46202] 2026-02-26T02:43:30.719 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43769] 2026-02-26T02:43:30.719 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46170] 2026-02-26T02:43:30.719 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46196] 2026-02-26T02:43:30.719 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45349] 2026-02-26T02:43:30.719 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46205] 2026-02-26T02:43:30.719 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46200] 2026-02-26T02:43:30.719 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46176] 2026-02-26T02:43:30.719 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46201] 2026-02-26T02:43:30.719 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46172] 2026-02-26T02:43:30.719 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46167] 2026-02-26T02:43:30.719 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46197] 2026-02-26T02:43:30.720 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46195] 2026-02-26T02:43:30.720 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46173] 2026-02-26T02:43:30.720 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46178] 2026-02-26T02:43:30.720 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46175] 2026-02-26T02:43:30.720 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46177] 2026-02-26T02:43:30.720 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46179] 2026-02-26T02:43:30.720 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46180] 2026-02-26T02:43:30.720 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46181] 2026-02-26T02:43:30.720 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46182] 2026-02-26T02:43:30.720 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46204] 2026-02-26T02:43:30.720 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46208] 2026-02-26T02:43:30.720 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45986] 2026-02-26T02:43:30.720 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45350] 2026-02-26T02:43:30.721 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46171] 2026-02-26T02:43:30.721 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46210] 2026-02-26T02:43:30.721 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46209] 2026-02-26T02:43:30.721 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46174] 2026-02-26T02:43:30.721 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46194] 2026-02-26T02:43:30.721 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46188] 2026-02-26T02:43:30.721 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46183] 2026-02-26T02:43:30.721 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46184] 2026-02-26T02:43:30.721 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46185] 2026-02-26T02:43:30.721 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46186] 2026-02-26T02:43:30.721 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46187] 2026-02-26T02:43:30.721 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46169] 2026-02-26T02:43:30.722 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46206] 2026-02-26T02:43:30.722 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46199] 2026-02-26T02:43:30.722 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46189] 2026-02-26T02:43:30.722 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46190] 2026-02-26T02:43:30.722 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46191] 2026-02-26T02:43:30.722 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46193] 2026-02-26T02:43:30.722 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46198] 2026-02-26T02:43:30.722 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46203] 2026-02-26T02:43:30.722 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46207] 2026-02-26T02:43:30.722 INFO:tasks.workunit.client.0.trial043.stderr:[Thread debugging using libthread_db enabled] 2026-02-26T02:43:30.722 INFO:tasks.workunit.client.0.trial043.stderr:Using host libthread_db library "/lib64/libthread_db.so.1". 2026-02-26T02:43:30.722 INFO:tasks.workunit.client.0.trial043.stderr:Core was generated by `/bin/ceph-osd -f --cluster ceph -i 1'\''. 2026-02-26T02:43:30.722 INFO:tasks.workunit.client.0.trial043.stderr:Program terminated with signal SIGABRT, Aborted. 2026-02-26T02:43:30.723 INFO:tasks.workunit.client.0.trial043.stderr:#0 0x00007f7ae908d03c in __pthread_kill_implementation () 2026-02-26T02:43:30.723 INFO:tasks.workunit.client.0.trial043.stderr: from /lib64/libc.so.6 2026-02-26T02:43:30.723 INFO:tasks.workunit.client.0.trial043.stderr:[Current thread is 1 (Thread 0x7f7ae8bf5780 (LWP 43516))] 2026-02-26T02:43:30.723 INFO:tasks.workunit.client.0.trial043.stderr:Missing rpms, try: dnf --enablerepo='\''*debug*'\'' install ceph-osd-debuginfo-19.2.3-687.g8d724284.el9.x86_64 gperftools-libs-debuginfo-2.9.1-3.el9.x86_64 libblkid-debuginfo-2.37.4-21.el9.x86_64 fuse-libs-debuginfo-2.9.9-17.el9.x86_64 libaio-debuginfo-0.3.111-13.el9.x86_64 snappy-debuginfo-1.1.8-8.el9.x86_64 lz4-libs-debuginfo-1.9.3-5.el9.x86_64 zlib-debuginfo-1.2.11-41.el9.x86_64 libcurl-debuginfo-7.76.1-40.el9.x86_64 thrift-debuginfo-0.15.0-4.el9.x86_64 libcap-debuginfo-2.48-10.el9.x86_64 glibc-debuginfo-2.34-245.el9.x86_64 openssl-libs-debuginfo-3.5.5-1.el9.x86_64 systemd-libs-debuginfo-252-64.el9.x86_64 libibverbs-debuginfo-61.0-2.el9.x86_64 librdmacm-debuginfo-61.0-2.el9.x86_64 libstdc++-debuginfo-11.5.0-14.el9.x86_64 libgcc-debuginfo-11.5.0-14.el9.x86_64 libunwind-debuginfo-1.6.2-1.el9.x86_64 libnghttp2-debuginfo-1.43.0-6.el9.x86_64 libidn2-debuginfo-2.3.0-7.el9.x86_64 libssh-debuginfo-0.10.4-17.el9.x86_64 libpsl-debuginfo-0.21.1-5.el9.x86_64 krb5-libs-debuginfo-1.21.1-8.el9.x86_64 libcom_err-debuginfo-1.46.5-8.el9.x86_64 openldap-debuginfo-2.6.8-4.el9.x86_64 libbrotli-debuginfo-1.0.9-9.el9.x86_64 libnl3-debuginfo-3.11.0-1.el9.x86_64 libunistring-debuginfo-0.9.10-15.el9.x86_64 keyutils-libs-debuginfo-1.6.3-1.el9.x86_64 libevent-debuginfo-2.1.12-8.el9.x86_64 cyrus-sasl-lib-debuginfo-2.1.27-21.el9.x86_64 libselinux-debuginfo-3.6-3.el9.x86_64 libxcrypt-debuginfo-4.4.18-3.el9.x86_64 pcre2-debuginfo-10.40-6.el9.x86_64 lttng-ust-debuginfo-2.12.0-6.el9.x86_64 numactl-libs-debuginfo-2.0.19-3.el9.x86_64 userspace-rcu-debuginfo-0.12.1-6.el9.x86_64 ceph-base-debuginfo-19.2.3-687.g8d724284.el9.x86_64 qatzip-libs-debuginfo-1.3.1-1.el9.x86_64 qatlib-debuginfo-25.08.0-2.el9.x86_64 liboath-debuginfo-2.6.12-1.el9.x86_64 2026-02-26T02:43:30.723 INFO:tasks.workunit.client.0.trial043.stderr:(gdb) ' '.*generated.*ceph-osd.*' 2026-02-26T02:43:30.723 INFO:tasks.workunit.client.0.trial043.stdout:4128 2026-02-26T02:43:30.724 INFO:tasks.workunit.client.0.trial043.stderr:+ expr match 'GNU gdb (CentOS Stream) 16.3-2.el9 2026-02-26T02:43:30.724 INFO:tasks.workunit.client.0.trial043.stderr:Copyright (C) 2024 Free Software Foundation, Inc. 2026-02-26T02:43:30.724 INFO:tasks.workunit.client.0.trial043.stderr:License GPLv3+: GNU GPL version 3 or later 2026-02-26T02:43:30.724 INFO:tasks.workunit.client.0.trial043.stderr:This is free software: you are free to change and redistribute it. 2026-02-26T02:43:30.724 INFO:tasks.workunit.client.0.trial043.stderr:There is NO WARRANTY, to the extent permitted by law. 2026-02-26T02:43:30.724 INFO:tasks.workunit.client.0.trial043.stderr:Type "show copying" and "show warranty" for details. 2026-02-26T02:43:30.725 INFO:tasks.workunit.client.0.trial043.stderr:This GDB was configured as "x86_64-redhat-linux-gnu". 2026-02-26T02:43:30.725 INFO:tasks.workunit.client.0.trial043.stderr:Type "show configuration" for configuration details. 2026-02-26T02:43:30.725 INFO:tasks.workunit.client.0.trial043.stderr:For bug reporting instructions, please see: 2026-02-26T02:43:30.725 INFO:tasks.workunit.client.0.trial043.stderr:. 2026-02-26T02:43:30.725 INFO:tasks.workunit.client.0.trial043.stderr:Find the GDB manual and other documentation resources online at: 2026-02-26T02:43:30.725 INFO:tasks.workunit.client.0.trial043.stderr: . 2026-02-26T02:43:30.725 INFO:tasks.workunit.client.0.trial043.stderr: 2026-02-26T02:43:30.725 INFO:tasks.workunit.client.0.trial043.stderr:For help, type "help". 2026-02-26T02:43:30.725 INFO:tasks.workunit.client.0.trial043.stderr:Type "apropos word" to search for commands related to "word"... 2026-02-26T02:43:30.725 INFO:tasks.workunit.client.0.trial043.stderr:Reading symbols from /usr/bin/ceph-osd... 2026-02-26T02:43:30.726 INFO:tasks.workunit.client.0.trial043.stderr:Reading symbols from .gnu_debugdata for /usr/bin/ceph-osd... 2026-02-26T02:43:30.726 INFO:tasks.workunit.client.0.trial043.stderr:(No debugging symbols found in .gnu_debugdata for /usr/bin/ceph-osd) 2026-02-26T02:43:30.726 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43516] 2026-02-26T02:43:30.726 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43526] 2026-02-26T02:43:30.726 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43530] 2026-02-26T02:43:30.726 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43538] 2026-02-26T02:43:30.726 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43539] 2026-02-26T02:43:30.726 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43755] 2026-02-26T02:43:30.726 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43528] 2026-02-26T02:43:30.726 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43757] 2026-02-26T02:43:30.726 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43842] 2026-02-26T02:43:30.727 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43843] 2026-02-26T02:43:30.727 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43771] 2026-02-26T02:43:30.727 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43844] 2026-02-26T02:43:30.727 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43529] 2026-02-26T02:43:30.727 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43845] 2026-02-26T02:43:30.727 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43846] 2026-02-26T02:43:30.727 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43847] 2026-02-26T02:43:30.727 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44242] 2026-02-26T02:43:30.727 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43754] 2026-02-26T02:43:30.727 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44245] 2026-02-26T02:43:30.727 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45985] 2026-02-26T02:43:30.728 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44243] 2026-02-26T02:43:30.728 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44244] 2026-02-26T02:43:30.728 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46192] 2026-02-26T02:43:30.728 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46168] 2026-02-26T02:43:30.728 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46202] 2026-02-26T02:43:30.728 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43769] 2026-02-26T02:43:30.728 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46170] 2026-02-26T02:43:30.728 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46196] 2026-02-26T02:43:30.728 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45349] 2026-02-26T02:43:30.728 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46205] 2026-02-26T02:43:30.728 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46200] 2026-02-26T02:43:30.728 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46176] 2026-02-26T02:43:30.729 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46201] 2026-02-26T02:43:30.729 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46172] 2026-02-26T02:43:30.729 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46167] 2026-02-26T02:43:30.729 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46197] 2026-02-26T02:43:30.729 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46195] 2026-02-26T02:43:30.729 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46173] 2026-02-26T02:43:30.729 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46178] 2026-02-26T02:43:30.729 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46175] 2026-02-26T02:43:30.729 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46177] 2026-02-26T02:43:30.729 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46179] 2026-02-26T02:43:30.730 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46180] 2026-02-26T02:43:30.730 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46181] 2026-02-26T02:43:30.730 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46182] 2026-02-26T02:43:30.730 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46204] 2026-02-26T02:43:30.730 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46208] 2026-02-26T02:43:30.730 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45986] 2026-02-26T02:43:30.730 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45350] 2026-02-26T02:43:30.730 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46171] 2026-02-26T02:43:30.730 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46210] 2026-02-26T02:43:30.730 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46209] 2026-02-26T02:43:30.730 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46174] 2026-02-26T02:43:30.730 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46194] 2026-02-26T02:43:30.731 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46188] 2026-02-26T02:43:30.731 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46183] 2026-02-26T02:43:30.731 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46184] 2026-02-26T02:43:30.731 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46185] 2026-02-26T02:43:30.731 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46186] 2026-02-26T02:43:30.731 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46187] 2026-02-26T02:43:30.731 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46169] 2026-02-26T02:43:30.731 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46206] 2026-02-26T02:43:30.731 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46199] 2026-02-26T02:43:30.731 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46189] 2026-02-26T02:43:30.731 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46190] 2026-02-26T02:43:30.731 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46191] 2026-02-26T02:43:30.731 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46193] 2026-02-26T02:43:30.732 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46198] 2026-02-26T02:43:30.732 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46203] 2026-02-26T02:43:30.732 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46207] 2026-02-26T02:43:30.732 INFO:tasks.workunit.client.0.trial043.stderr:[Thread debugging using libthread_db enabled] 2026-02-26T02:43:30.732 INFO:tasks.workunit.client.0.trial043.stderr:Using host libthread_db library "/lib64/libthread_db.so.1". 2026-02-26T02:43:30.732 INFO:tasks.workunit.client.0.trial043.stderr:Core was generated by `/bin/ceph-osd -f --cluster ceph -i 1'\''. 2026-02-26T02:43:30.732 INFO:tasks.workunit.client.0.trial043.stderr:Program terminated with signal SIGABRT, Aborted. 2026-02-26T02:43:30.732 INFO:tasks.workunit.client.0.trial043.stderr:#0 0x00007f7ae908d03c in __pthread_kill_implementation () 2026-02-26T02:43:30.732 INFO:tasks.workunit.client.0.trial043.stderr: from /lib64/libc.so.6 2026-02-26T02:43:30.732 INFO:tasks.workunit.client.0.trial043.stderr:[Current thread is 1 (Thread 0x7f7ae8bf5780 (LWP 43516))] 2026-02-26T02:43:30.732 INFO:tasks.workunit.client.0.trial043.stderr:Missing rpms, try: dnf --enablerepo='\''*debug*'\'' install ceph-osd-debuginfo-19.2.3-687.g8d724284.el9.x86_64 gperftools-libs-debuginfo-2.9.1-3.el9.x86_64 libblkid-debuginfo-2.37.4-21.el9.x86_64 fuse-libs-debuginfo-2.9.9-17.el9.x86_64 libaio-debuginfo-0.3.111-13.el9.x86_64 snappy-debuginfo-1.1.8-8.el9.x86_64 lz4-libs-debuginfo-1.9.3-5.el9.x86_64 zlib-debuginfo-1.2.11-41.el9.x86_64 libcurl-debuginfo-7.76.1-40.el9.x86_64 thrift-debuginfo-0.15.0-4.el9.x86_64 libcap-debuginfo-2.48-10.el9.x86_64 glibc-debuginfo-2.34-245.el9.x86_64 openssl-libs-debuginfo-3.5.5-1.el9.x86_64 systemd-libs-debuginfo-252-64.el9.x86_64 libibverbs-debuginfo-61.0-2.el9.x86_64 librdmacm-debuginfo-61.0-2.el9.x86_64 libstdc++-debuginfo-11.5.0-14.el9.x86_64 libgcc-debuginfo-11.5.0-14.el9.x86_64 libunwind-debuginfo-1.6.2-1.el9.x86_64 libnghttp2-debuginfo-1.43.0-6.el9.x86_64 libidn2-debuginfo-2.3.0-7.el9.x86_64 libssh-debuginfo-0.10.4-17.el9.x86_64 libpsl-debuginfo-0.21.1-5.el9.x86_64 krb5-libs-debuginfo-1.21.1-8.el9.x86_64 libcom_err-debuginfo-1.46.5-8.el9.x86_64 openldap-debuginfo-2.6.8-4.el9.x86_64 libbrotli-debuginfo-1.0.9-9.el9.x86_64 libnl3-debuginfo-3.11.0-1.el9.x86_64 libunistring-debuginfo-0.9.10-15.el9.x86_64 keyutils-libs-debuginfo-1.6.3-1.el9.x86_64 libevent-debuginfo-2.1.12-8.el9.x86_64 cyrus-sasl-lib-debuginfo-2.1.27-21.el9.x86_64 libselinux-debuginfo-3.6-3.el9.x86_64 libxcrypt-debuginfo-4.4.18-3.el9.x86_64 pcre2-debuginfo-10.40-6.el9.x86_64 lttng-ust-debuginfo-2.12.0-6.el9.x86_64 numactl-libs-debuginfo-2.0.19-3.el9.x86_64 userspace-rcu-debuginfo-0.12.1-6.el9.x86_64 ceph-base-debuginfo-19.2.3-687.g8d724284.el9.x86_64 qatzip-libs-debuginfo-1.3.1-1.el9.x86_64 qatlib-debuginfo-25.08.0-2.el9.x86_64 liboath-debuginfo-2.6.12-1.el9.x86_64 2026-02-26T02:43:30.732 INFO:tasks.workunit.client.0.trial043.stderr:(gdb) ' '.*terminated.*signal 6.*' 2026-02-26T02:43:30.733 INFO:tasks.workunit.client.0.trial043.stdout:0 2026-02-26T02:43:30.733 INFO:tasks.workunit.client.0.trial043.stderr:+ expr match 'GNU gdb (CentOS Stream) 16.3-2.el9 2026-02-26T02:43:30.733 INFO:tasks.workunit.client.0.trial043.stderr:Copyright (C) 2024 Free Software Foundation, Inc. 2026-02-26T02:43:30.733 INFO:tasks.workunit.client.0.trial043.stderr:License GPLv3+: GNU GPL version 3 or later 2026-02-26T02:43:30.734 INFO:tasks.workunit.client.0.trial043.stderr:This is free software: you are free to change and redistribute it. 2026-02-26T02:43:30.734 INFO:tasks.workunit.client.0.trial043.stderr:There is NO WARRANTY, to the extent permitted by law. 2026-02-26T02:43:30.734 INFO:tasks.workunit.client.0.trial043.stderr:Type "show copying" and "show warranty" for details. 2026-02-26T02:43:30.734 INFO:tasks.workunit.client.0.trial043.stderr:This GDB was configured as "x86_64-redhat-linux-gnu". 2026-02-26T02:43:30.734 INFO:tasks.workunit.client.0.trial043.stderr:Type "show configuration" for configuration details. 2026-02-26T02:43:30.734 INFO:tasks.workunit.client.0.trial043.stderr:For bug reporting instructions, please see: 2026-02-26T02:43:30.734 INFO:tasks.workunit.client.0.trial043.stderr:. 2026-02-26T02:43:30.734 INFO:tasks.workunit.client.0.trial043.stderr:Find the GDB manual and other documentation resources online at: 2026-02-26T02:43:30.734 INFO:tasks.workunit.client.0.trial043.stderr: . 2026-02-26T02:43:30.734 INFO:tasks.workunit.client.0.trial043.stderr: 2026-02-26T02:43:30.734 INFO:tasks.workunit.client.0.trial043.stderr:For help, type "help". 2026-02-26T02:43:30.735 INFO:tasks.workunit.client.0.trial043.stderr:Type "apropos word" to search for commands related to "word"... 2026-02-26T02:43:30.735 INFO:tasks.workunit.client.0.trial043.stderr:Reading symbols from /usr/bin/ceph-osd... 2026-02-26T02:43:30.735 INFO:tasks.workunit.client.0.trial043.stderr:Reading symbols from .gnu_debugdata for /usr/bin/ceph-osd... 2026-02-26T02:43:30.735 INFO:tasks.workunit.client.0.trial043.stderr:(No debugging symbols found in .gnu_debugdata for /usr/bin/ceph-osd) 2026-02-26T02:43:30.735 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43516] 2026-02-26T02:43:30.735 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43526] 2026-02-26T02:43:30.735 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43530] 2026-02-26T02:43:30.735 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43538] 2026-02-26T02:43:30.735 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43539] 2026-02-26T02:43:30.735 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43755] 2026-02-26T02:43:30.735 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43528] 2026-02-26T02:43:30.736 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43757] 2026-02-26T02:43:30.736 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43842] 2026-02-26T02:43:30.736 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43843] 2026-02-26T02:43:30.736 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43771] 2026-02-26T02:43:30.736 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43844] 2026-02-26T02:43:30.736 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43529] 2026-02-26T02:43:30.736 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43845] 2026-02-26T02:43:30.736 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43846] 2026-02-26T02:43:30.736 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43847] 2026-02-26T02:43:30.736 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44242] 2026-02-26T02:43:30.736 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43754] 2026-02-26T02:43:30.737 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44245] 2026-02-26T02:43:30.737 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45985] 2026-02-26T02:43:30.737 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44243] 2026-02-26T02:43:30.737 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44244] 2026-02-26T02:43:30.737 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46192] 2026-02-26T02:43:30.737 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46168] 2026-02-26T02:43:30.737 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46202] 2026-02-26T02:43:30.737 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43769] 2026-02-26T02:43:30.737 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46170] 2026-02-26T02:43:30.737 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46196] 2026-02-26T02:43:30.737 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45349] 2026-02-26T02:43:30.738 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46205] 2026-02-26T02:43:30.738 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46200] 2026-02-26T02:43:30.738 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46176] 2026-02-26T02:43:30.738 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46201] 2026-02-26T02:43:30.738 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46172] 2026-02-26T02:43:30.738 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46167] 2026-02-26T02:43:30.738 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46197] 2026-02-26T02:43:30.738 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46195] 2026-02-26T02:43:30.738 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46173] 2026-02-26T02:43:30.738 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46178] 2026-02-26T02:43:30.738 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46175] 2026-02-26T02:43:30.738 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46177] 2026-02-26T02:43:30.738 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46179] 2026-02-26T02:43:30.739 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46180] 2026-02-26T02:43:30.739 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46181] 2026-02-26T02:43:30.739 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46182] 2026-02-26T02:43:30.739 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46204] 2026-02-26T02:43:30.739 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46208] 2026-02-26T02:43:30.739 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45986] 2026-02-26T02:43:30.739 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45350] 2026-02-26T02:43:30.739 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46171] 2026-02-26T02:43:30.739 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46210] 2026-02-26T02:43:30.739 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46209] 2026-02-26T02:43:30.739 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46174] 2026-02-26T02:43:30.739 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46194] 2026-02-26T02:43:30.739 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46188] 2026-02-26T02:43:30.740 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46183] 2026-02-26T02:43:30.740 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46184] 2026-02-26T02:43:30.740 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46185] 2026-02-26T02:43:30.740 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46186] 2026-02-26T02:43:30.740 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46187] 2026-02-26T02:43:30.740 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46169] 2026-02-26T02:43:30.740 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46206] 2026-02-26T02:43:30.740 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46199] 2026-02-26T02:43:30.740 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46189] 2026-02-26T02:43:30.740 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46190] 2026-02-26T02:43:30.740 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46191] 2026-02-26T02:43:30.740 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46193] 2026-02-26T02:43:30.740 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46198] 2026-02-26T02:43:30.741 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46203] 2026-02-26T02:43:30.741 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 46207] 2026-02-26T02:43:30.741 INFO:tasks.workunit.client.0.trial043.stderr:[Thread debugging using libthread_db enabled] 2026-02-26T02:43:30.741 INFO:tasks.workunit.client.0.trial043.stderr:Using host libthread_db library "/lib64/libthread_db.so.1". 2026-02-26T02:43:30.741 INFO:tasks.workunit.client.0.trial043.stderr:Core was generated by `/bin/ceph-osd -f --cluster ceph -i 1'\''. 2026-02-26T02:43:30.741 INFO:tasks.workunit.client.0.trial043.stderr:Program terminated with signal SIGABRT, Aborted. 2026-02-26T02:43:30.741 INFO:tasks.workunit.client.0.trial043.stderr:#0 0x00007f7ae908d03c in __pthread_kill_implementation () 2026-02-26T02:43:30.741 INFO:tasks.workunit.client.0.trial043.stderr: from /lib64/libc.so.6 2026-02-26T02:43:30.741 INFO:tasks.workunit.client.0.trial043.stderr:[Current thread is 1 (Thread 0x7f7ae8bf5780 (LWP 43516))] 2026-02-26T02:43:30.741 INFO:tasks.workunit.client.0.trial043.stderr:Missing rpms, try: dnf --enablerepo='\''*debug*'\'' install ceph-osd-debuginfo-19.2.3-687.g8d724284.el9.x86_64 gperftools-libs-debuginfo-2.9.1-3.el9.x86_64 libblkid-debuginfo-2.37.4-21.el9.x86_64 fuse-libs-debuginfo-2.9.9-17.el9.x86_64 libaio-debuginfo-0.3.111-13.el9.x86_64 snappy-debuginfo-1.1.8-8.el9.x86_64 lz4-libs-debuginfo-1.9.3-5.el9.x86_64 zlib-debuginfo-1.2.11-41.el9.x86_64 libcurl-debuginfo-7.76.1-40.el9.x86_64 thrift-debuginfo-0.15.0-4.el9.x86_64 libcap-debuginfo-2.48-10.el9.x86_64 glibc-debuginfo-2.34-245.el9.x86_64 openssl-libs-debuginfo-3.5.5-1.el9.x86_64 systemd-libs-debuginfo-252-64.el9.x86_64 libibverbs-debuginfo-61.0-2.el9.x86_64 librdmacm-debuginfo-61.0-2.el9.x86_64 libstdc++-debuginfo-11.5.0-14.el9.x86_64 libgcc-debuginfo-11.5.0-14.el9.x86_64 libunwind-debuginfo-1.6.2-1.el9.x86_64 libnghttp2-debuginfo-1.43.0-6.el9.x86_64 libidn2-debuginfo-2.3.0-7.el9.x86_64 libssh-debuginfo-0.10.4-17.el9.x86_64 libpsl-debuginfo-0.21.1-5.el9.x86_64 krb5-libs-debuginfo-1.21.1-8.el9.x86_64 libcom_err-debuginfo-1.46.5-8.el9.x86_64 openldap-debuginfo-2.6.8-4.el9.x86_64 libbrotli-debuginfo-1.0.9-9.el9.x86_64 libnl3-debuginfo-3.11.0-1.el9.x86_64 libunistring-debuginfo-0.9.10-15.el9.x86_64 keyutils-libs-debuginfo-1.6.3-1.el9.x86_64 libevent-debuginfo-2.1.12-8.el9.x86_64 cyrus-sasl-lib-debuginfo-2.1.27-21.el9.x86_64 libselinux-debuginfo-3.6-3.el9.x86_64 libxcrypt-debuginfo-4.4.18-3.el9.x86_64 pcre2-debuginfo-10.40-6.el9.x86_64 lttng-ust-debuginfo-2.12.0-6.el9.x86_64 numactl-libs-debuginfo-2.0.19-3.el9.x86_64 userspace-rcu-debuginfo-0.12.1-6.el9.x86_64 ceph-base-debuginfo-19.2.3-687.g8d724284.el9.x86_64 qatzip-libs-debuginfo-1.3.1-1.el9.x86_64 qatlib-debuginfo-25.08.0-2.el9.x86_64 liboath-debuginfo-2.6.12-1.el9.x86_64 2026-02-26T02:43:30.741 INFO:tasks.workunit.client.0.trial043.stderr:(gdb) ' '.*terminated.*signal SIGABRT.*' 2026-02-26T02:43:30.742 INFO:tasks.workunit.client.0.trial043.stdout:4128 2026-02-26T02:43:30.742 INFO:tasks.workunit.client.0.trial043.stderr:+ sudo rm /home/ubuntu/cephtest/archive/coredump/1772073805.43516.core 2026-02-26T02:43:30.742 INFO:tasks.workunit.client.0.trial043.stderr:+ for f in $(find $TESTDIR/archive/coredump -type f) 2026-02-26T02:43:30.742 INFO:tasks.workunit.client.0.trial043.stderr:++ echo quit 2026-02-26T02:43:30.742 INFO:tasks.workunit.client.0.trial043.stderr:++ sudo gdb /usr/bin/ceph-osd /home/ubuntu/cephtest/archive/coredump/1772073805.43515.core 2026-02-26T02:43:30.825 INFO:tasks.workunit.client.0.trial043.stderr: 2026-02-26T02:43:30.825 INFO:tasks.workunit.client.0.trial043.stderr:warning: Can't open file /[aio] (deleted) during file-backed mapping note processing 2026-02-26T02:43:31.000 INFO:tasks.workunit.client.0.trial043.stderr:+ gdb_output='GNU gdb (CentOS Stream) 16.3-2.el9 2026-02-26T02:43:31.000 INFO:tasks.workunit.client.0.trial043.stderr:Copyright (C) 2024 Free Software Foundation, Inc. 2026-02-26T02:43:31.000 INFO:tasks.workunit.client.0.trial043.stderr:License GPLv3+: GNU GPL version 3 or later 2026-02-26T02:43:31.000 INFO:tasks.workunit.client.0.trial043.stderr:This is free software: you are free to change and redistribute it. 2026-02-26T02:43:31.000 INFO:tasks.workunit.client.0.trial043.stderr:There is NO WARRANTY, to the extent permitted by law. 2026-02-26T02:43:31.001 INFO:tasks.workunit.client.0.trial043.stderr:Type "show copying" and "show warranty" for details. 2026-02-26T02:43:31.001 INFO:tasks.workunit.client.0.trial043.stderr:This GDB was configured as "x86_64-redhat-linux-gnu". 2026-02-26T02:43:31.001 INFO:tasks.workunit.client.0.trial043.stderr:Type "show configuration" for configuration details. 2026-02-26T02:43:31.001 INFO:tasks.workunit.client.0.trial043.stderr:For bug reporting instructions, please see: 2026-02-26T02:43:31.001 INFO:tasks.workunit.client.0.trial043.stderr:. 2026-02-26T02:43:31.001 INFO:tasks.workunit.client.0.trial043.stderr:Find the GDB manual and other documentation resources online at: 2026-02-26T02:43:31.001 INFO:tasks.workunit.client.0.trial043.stderr: . 2026-02-26T02:43:31.001 INFO:tasks.workunit.client.0.trial043.stderr: 2026-02-26T02:43:31.001 INFO:tasks.workunit.client.0.trial043.stderr:For help, type "help". 2026-02-26T02:43:31.001 INFO:tasks.workunit.client.0.trial043.stderr:Type "apropos word" to search for commands related to "word"... 2026-02-26T02:43:31.001 INFO:tasks.workunit.client.0.trial043.stderr:Reading symbols from /usr/bin/ceph-osd... 2026-02-26T02:43:31.002 INFO:tasks.workunit.client.0.trial043.stderr:Reading symbols from .gnu_debugdata for /usr/bin/ceph-osd... 2026-02-26T02:43:31.002 INFO:tasks.workunit.client.0.trial043.stderr:(No debugging symbols found in .gnu_debugdata for /usr/bin/ceph-osd) 2026-02-26T02:43:31.002 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43515] 2026-02-26T02:43:31.002 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43520] 2026-02-26T02:43:31.002 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43522] 2026-02-26T02:43:31.002 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43518] 2026-02-26T02:43:31.002 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43535] 2026-02-26T02:43:31.002 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43759] 2026-02-26T02:43:31.002 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43758] 2026-02-26T02:43:31.002 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43521] 2026-02-26T02:43:31.002 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43768] 2026-02-26T02:43:31.002 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43835] 2026-02-26T02:43:31.002 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43836] 2026-02-26T02:43:31.003 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43756] 2026-02-26T02:43:31.003 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43783] 2026-02-26T02:43:31.003 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43790] 2026-02-26T02:43:31.003 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43534] 2026-02-26T02:43:31.003 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44054] 2026-02-26T02:43:31.003 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45166] 2026-02-26T02:43:31.003 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45759] 2026-02-26T02:43:31.003 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45760] 2026-02-26T02:43:31.003 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45943] 2026-02-26T02:43:31.003 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43797] 2026-02-26T02:43:31.003 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45941] 2026-02-26T02:43:31.003 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43770] 2026-02-26T02:43:31.003 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43778] 2026-02-26T02:43:31.004 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45958] 2026-02-26T02:43:31.004 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45950] 2026-02-26T02:43:31.004 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45968] 2026-02-26T02:43:31.004 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45952] 2026-02-26T02:43:31.004 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45948] 2026-02-26T02:43:31.004 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45969] 2026-02-26T02:43:31.004 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43834] 2026-02-26T02:43:31.004 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45972] 2026-02-26T02:43:31.004 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45984] 2026-02-26T02:43:31.004 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45974] 2026-02-26T02:43:31.004 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43804] 2026-02-26T02:43:31.004 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44055] 2026-02-26T02:43:31.004 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45946] 2026-02-26T02:43:31.005 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45973] 2026-02-26T02:43:31.005 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45966] 2026-02-26T02:43:31.005 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45971] 2026-02-26T02:43:31.005 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45954] 2026-02-26T02:43:31.005 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45980] 2026-02-26T02:43:31.005 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45947] 2026-02-26T02:43:31.005 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45951] 2026-02-26T02:43:31.005 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43813] 2026-02-26T02:43:31.005 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45942] 2026-02-26T02:43:31.005 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45967] 2026-02-26T02:43:31.005 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45944] 2026-02-26T02:43:31.005 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45945] 2026-02-26T02:43:31.005 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44056] 2026-02-26T02:43:31.006 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45975] 2026-02-26T02:43:31.006 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44057] 2026-02-26T02:43:31.006 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45953] 2026-02-26T02:43:31.006 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45956] 2026-02-26T02:43:31.006 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45976] 2026-02-26T02:43:31.006 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45978] 2026-02-26T02:43:31.006 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43820] 2026-02-26T02:43:31.006 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45955] 2026-02-26T02:43:31.006 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45981] 2026-02-26T02:43:31.006 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45957] 2026-02-26T02:43:31.006 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45983] 2026-02-26T02:43:31.006 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45959] 2026-02-26T02:43:31.006 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45982] 2026-02-26T02:43:31.007 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43833] 2026-02-26T02:43:31.007 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43837] 2026-02-26T02:43:31.007 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45949] 2026-02-26T02:43:31.007 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45960] 2026-02-26T02:43:31.007 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45961] 2026-02-26T02:43:31.007 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43838] 2026-02-26T02:43:31.007 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45962] 2026-02-26T02:43:31.007 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45963] 2026-02-26T02:43:31.007 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45964] 2026-02-26T02:43:31.007 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43839] 2026-02-26T02:43:31.007 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45970] 2026-02-26T02:43:31.007 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45165] 2026-02-26T02:43:31.007 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45977] 2026-02-26T02:43:31.008 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45979] 2026-02-26T02:43:31.008 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45965] 2026-02-26T02:43:31.008 INFO:tasks.workunit.client.0.trial043.stderr:[Thread debugging using libthread_db enabled] 2026-02-26T02:43:31.008 INFO:tasks.workunit.client.0.trial043.stderr:Using host libthread_db library "/lib64/libthread_db.so.1". 2026-02-26T02:43:31.008 INFO:tasks.workunit.client.0.trial043.stderr:Core was generated by `/bin/ceph-osd -f --cluster ceph -i 0'\''. 2026-02-26T02:43:31.008 INFO:tasks.workunit.client.0.trial043.stderr:Program terminated with signal SIGABRT, Aborted. 2026-02-26T02:43:31.008 INFO:tasks.workunit.client.0.trial043.stderr:#0 0x00007ff7a308d03c in __pthread_kill_implementation () 2026-02-26T02:43:31.008 INFO:tasks.workunit.client.0.trial043.stderr: from /lib64/libc.so.6 2026-02-26T02:43:31.008 INFO:tasks.workunit.client.0.trial043.stderr:[Current thread is 1 (Thread 0x7ff7a2b34780 (LWP 43515))] 2026-02-26T02:43:31.008 INFO:tasks.workunit.client.0.trial043.stderr:Missing rpms, try: dnf --enablerepo='\''*debug*'\'' install ceph-osd-debuginfo-19.2.3-687.g8d724284.el9.x86_64 gperftools-libs-debuginfo-2.9.1-3.el9.x86_64 libblkid-debuginfo-2.37.4-21.el9.x86_64 fuse-libs-debuginfo-2.9.9-17.el9.x86_64 libaio-debuginfo-0.3.111-13.el9.x86_64 snappy-debuginfo-1.1.8-8.el9.x86_64 lz4-libs-debuginfo-1.9.3-5.el9.x86_64 zlib-debuginfo-1.2.11-41.el9.x86_64 libcurl-debuginfo-7.76.1-40.el9.x86_64 thrift-debuginfo-0.15.0-4.el9.x86_64 libcap-debuginfo-2.48-10.el9.x86_64 glibc-debuginfo-2.34-245.el9.x86_64 openssl-libs-debuginfo-3.5.5-1.el9.x86_64 systemd-libs-debuginfo-252-64.el9.x86_64 libibverbs-debuginfo-61.0-2.el9.x86_64 librdmacm-debuginfo-61.0-2.el9.x86_64 libstdc++-debuginfo-11.5.0-14.el9.x86_64 libgcc-debuginfo-11.5.0-14.el9.x86_64 libunwind-debuginfo-1.6.2-1.el9.x86_64 libnghttp2-debuginfo-1.43.0-6.el9.x86_64 libidn2-debuginfo-2.3.0-7.el9.x86_64 libssh-debuginfo-0.10.4-17.el9.x86_64 libpsl-debuginfo-0.21.1-5.el9.x86_64 krb5-libs-debuginfo-1.21.1-8.el9.x86_64 libcom_err-debuginfo-1.46.5-8.el9.x86_64 openldap-debuginfo-2.6.8-4.el9.x86_64 libbrotli-debuginfo-1.0.9-9.el9.x86_64 libnl3-debuginfo-3.11.0-1.el9.x86_64 libunistring-debuginfo-0.9.10-15.el9.x86_64 keyutils-libs-debuginfo-1.6.3-1.el9.x86_64 libevent-debuginfo-2.1.12-8.el9.x86_64 cyrus-sasl-lib-debuginfo-2.1.27-21.el9.x86_64 libselinux-debuginfo-3.6-3.el9.x86_64 libxcrypt-debuginfo-4.4.18-3.el9.x86_64 pcre2-debuginfo-10.40-6.el9.x86_64 lttng-ust-debuginfo-2.12.0-6.el9.x86_64 numactl-libs-debuginfo-2.0.19-3.el9.x86_64 userspace-rcu-debuginfo-0.12.1-6.el9.x86_64 ceph-base-debuginfo-19.2.3-687.g8d724284.el9.x86_64 qatzip-libs-debuginfo-1.3.1-1.el9.x86_64 qatlib-debuginfo-25.08.0-2.el9.x86_64 liboath-debuginfo-2.6.12-1.el9.x86_64 2026-02-26T02:43:31.009 INFO:tasks.workunit.client.0.trial043.stderr:(gdb) ' 2026-02-26T02:43:31.009 INFO:tasks.workunit.client.0.trial043.stderr:+ expr match 'GNU gdb (CentOS Stream) 16.3-2.el9 2026-02-26T02:43:31.009 INFO:tasks.workunit.client.0.trial043.stderr:Copyright (C) 2024 Free Software Foundation, Inc. 2026-02-26T02:43:31.009 INFO:tasks.workunit.client.0.trial043.stderr:License GPLv3+: GNU GPL version 3 or later 2026-02-26T02:43:31.009 INFO:tasks.workunit.client.0.trial043.stderr:This is free software: you are free to change and redistribute it. 2026-02-26T02:43:31.009 INFO:tasks.workunit.client.0.trial043.stderr:There is NO WARRANTY, to the extent permitted by law. 2026-02-26T02:43:31.009 INFO:tasks.workunit.client.0.trial043.stderr:Type "show copying" and "show warranty" for details. 2026-02-26T02:43:31.009 INFO:tasks.workunit.client.0.trial043.stderr:This GDB was configured as "x86_64-redhat-linux-gnu". 2026-02-26T02:43:31.009 INFO:tasks.workunit.client.0.trial043.stderr:Type "show configuration" for configuration details. 2026-02-26T02:43:31.009 INFO:tasks.workunit.client.0.trial043.stderr:For bug reporting instructions, please see: 2026-02-26T02:43:31.009 INFO:tasks.workunit.client.0.trial043.stderr:. 2026-02-26T02:43:31.010 INFO:tasks.workunit.client.0.trial043.stderr:Find the GDB manual and other documentation resources online at: 2026-02-26T02:43:31.010 INFO:tasks.workunit.client.0.trial043.stderr: . 2026-02-26T02:43:31.010 INFO:tasks.workunit.client.0.trial043.stderr: 2026-02-26T02:43:31.010 INFO:tasks.workunit.client.0.trial043.stderr:For help, type "help". 2026-02-26T02:43:31.010 INFO:tasks.workunit.client.0.trial043.stderr:Type "apropos word" to search for commands related to "word"... 2026-02-26T02:43:31.010 INFO:tasks.workunit.client.0.trial043.stderr:Reading symbols from /usr/bin/ceph-osd... 2026-02-26T02:43:31.010 INFO:tasks.workunit.client.0.trial043.stderr:Reading symbols from .gnu_debugdata for /usr/bin/ceph-osd... 2026-02-26T02:43:31.010 INFO:tasks.workunit.client.0.trial043.stderr:(No debugging symbols found in .gnu_debugdata for /usr/bin/ceph-osd) 2026-02-26T02:43:31.010 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43515] 2026-02-26T02:43:31.010 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43520] 2026-02-26T02:43:31.011 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43522] 2026-02-26T02:43:31.011 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43518] 2026-02-26T02:43:31.011 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43535] 2026-02-26T02:43:31.011 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43759] 2026-02-26T02:43:31.011 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43758] 2026-02-26T02:43:31.011 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43521] 2026-02-26T02:43:31.011 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43768] 2026-02-26T02:43:31.011 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43835] 2026-02-26T02:43:31.011 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43836] 2026-02-26T02:43:31.011 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43756] 2026-02-26T02:43:31.011 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43783] 2026-02-26T02:43:31.012 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43790] 2026-02-26T02:43:31.012 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43534] 2026-02-26T02:43:31.012 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44054] 2026-02-26T02:43:31.012 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45166] 2026-02-26T02:43:31.012 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45759] 2026-02-26T02:43:31.012 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45760] 2026-02-26T02:43:31.012 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45943] 2026-02-26T02:43:31.012 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43797] 2026-02-26T02:43:31.012 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45941] 2026-02-26T02:43:31.012 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43770] 2026-02-26T02:43:31.012 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43778] 2026-02-26T02:43:31.012 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45958] 2026-02-26T02:43:31.013 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45950] 2026-02-26T02:43:31.013 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45968] 2026-02-26T02:43:31.013 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45952] 2026-02-26T02:43:31.013 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45948] 2026-02-26T02:43:31.013 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45969] 2026-02-26T02:43:31.013 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43834] 2026-02-26T02:43:31.013 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45972] 2026-02-26T02:43:31.013 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45984] 2026-02-26T02:43:31.013 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45974] 2026-02-26T02:43:31.013 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43804] 2026-02-26T02:43:31.013 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44055] 2026-02-26T02:43:31.014 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45946] 2026-02-26T02:43:31.014 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45973] 2026-02-26T02:43:31.014 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45966] 2026-02-26T02:43:31.014 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45971] 2026-02-26T02:43:31.014 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45954] 2026-02-26T02:43:31.014 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45980] 2026-02-26T02:43:31.014 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45947] 2026-02-26T02:43:31.014 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45951] 2026-02-26T02:43:31.014 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43813] 2026-02-26T02:43:31.014 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45942] 2026-02-26T02:43:31.014 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45967] 2026-02-26T02:43:31.015 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45944] 2026-02-26T02:43:31.015 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45945] 2026-02-26T02:43:31.015 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44056] 2026-02-26T02:43:31.015 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45975] 2026-02-26T02:43:31.015 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44057] 2026-02-26T02:43:31.015 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45953] 2026-02-26T02:43:31.015 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45956] 2026-02-26T02:43:31.015 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45976] 2026-02-26T02:43:31.015 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45978] 2026-02-26T02:43:31.015 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43820] 2026-02-26T02:43:31.016 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45955] 2026-02-26T02:43:31.016 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45981] 2026-02-26T02:43:31.016 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45957] 2026-02-26T02:43:31.016 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45983] 2026-02-26T02:43:31.016 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45959] 2026-02-26T02:43:31.016 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45982] 2026-02-26T02:43:31.016 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43833] 2026-02-26T02:43:31.016 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43837] 2026-02-26T02:43:31.016 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45949] 2026-02-26T02:43:31.016 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45960] 2026-02-26T02:43:31.016 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45961] 2026-02-26T02:43:31.016 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43838] 2026-02-26T02:43:31.017 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45962] 2026-02-26T02:43:31.017 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45963] 2026-02-26T02:43:31.017 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45964] 2026-02-26T02:43:31.017 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43839] 2026-02-26T02:43:31.017 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45970] 2026-02-26T02:43:31.017 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45165] 2026-02-26T02:43:31.017 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45977] 2026-02-26T02:43:31.017 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45979] 2026-02-26T02:43:31.017 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45965] 2026-02-26T02:43:31.017 INFO:tasks.workunit.client.0.trial043.stderr:[Thread debugging using libthread_db enabled] 2026-02-26T02:43:31.017 INFO:tasks.workunit.client.0.trial043.stderr:Using host libthread_db library "/lib64/libthread_db.so.1". 2026-02-26T02:43:31.018 INFO:tasks.workunit.client.0.trial043.stderr:Core was generated by `/bin/ceph-osd -f --cluster ceph -i 0'\''. 2026-02-26T02:43:31.018 INFO:tasks.workunit.client.0.trial043.stderr:Program terminated with signal SIGABRT, Aborted. 2026-02-26T02:43:31.018 INFO:tasks.workunit.client.0.trial043.stderr:#0 0x00007ff7a308d03c in __pthread_kill_implementation () 2026-02-26T02:43:31.018 INFO:tasks.workunit.client.0.trial043.stderr: from /lib64/libc.so.6 2026-02-26T02:43:31.018 INFO:tasks.workunit.client.0.trial043.stderr:[Current thread is 1 (Thread 0x7ff7a2b34780 (LWP 43515))] 2026-02-26T02:43:31.018 INFO:tasks.workunit.client.0.trial043.stderr:Missing rpms, try: dnf --enablerepo='\''*debug*'\'' install ceph-osd-debuginfo-19.2.3-687.g8d724284.el9.x86_64 gperftools-libs-debuginfo-2.9.1-3.el9.x86_64 libblkid-debuginfo-2.37.4-21.el9.x86_64 fuse-libs-debuginfo-2.9.9-17.el9.x86_64 libaio-debuginfo-0.3.111-13.el9.x86_64 snappy-debuginfo-1.1.8-8.el9.x86_64 lz4-libs-debuginfo-1.9.3-5.el9.x86_64 zlib-debuginfo-1.2.11-41.el9.x86_64 libcurl-debuginfo-7.76.1-40.el9.x86_64 thrift-debuginfo-0.15.0-4.el9.x86_64 libcap-debuginfo-2.48-10.el9.x86_64 glibc-debuginfo-2.34-245.el9.x86_64 openssl-libs-debuginfo-3.5.5-1.el9.x86_64 systemd-libs-debuginfo-252-64.el9.x86_64 libibverbs-debuginfo-61.0-2.el9.x86_64 librdmacm-debuginfo-61.0-2.el9.x86_64 libstdc++-debuginfo-11.5.0-14.el9.x86_64 libgcc-debuginfo-11.5.0-14.el9.x86_64 libunwind-debuginfo-1.6.2-1.el9.x86_64 libnghttp2-debuginfo-1.43.0-6.el9.x86_64 libidn2-debuginfo-2.3.0-7.el9.x86_64 libssh-debuginfo-0.10.4-17.el9.x86_64 libpsl-debuginfo-0.21.1-5.el9.x86_64 krb5-libs-debuginfo-1.21.1-8.el9.x86_64 libcom_err-debuginfo-1.46.5-8.el9.x86_64 openldap-debuginfo-2.6.8-4.el9.x86_64 libbrotli-debuginfo-1.0.9-9.el9.x86_64 libnl3-debuginfo-3.11.0-1.el9.x86_64 libunistring-debuginfo-0.9.10-15.el9.x86_64 keyutils-libs-debuginfo-1.6.3-1.el9.x86_64 libevent-debuginfo-2.1.12-8.el9.x86_64 cyrus-sasl-lib-debuginfo-2.1.27-21.el9.x86_64 libselinux-debuginfo-3.6-3.el9.x86_64 libxcrypt-debuginfo-4.4.18-3.el9.x86_64 pcre2-debuginfo-10.40-6.el9.x86_64 lttng-ust-debuginfo-2.12.0-6.el9.x86_64 numactl-libs-debuginfo-2.0.19-3.el9.x86_64 userspace-rcu-debuginfo-0.12.1-6.el9.x86_64 ceph-base-debuginfo-19.2.3-687.g8d724284.el9.x86_64 qatzip-libs-debuginfo-1.3.1-1.el9.x86_64 qatlib-debuginfo-25.08.0-2.el9.x86_64 liboath-debuginfo-2.6.12-1.el9.x86_64 2026-02-26T02:43:31.018 INFO:tasks.workunit.client.0.trial043.stderr:(gdb) ' '.*generated.*ceph-osd.*' 2026-02-26T02:43:31.019 INFO:tasks.workunit.client.0.trial043.stdout:4256 2026-02-26T02:43:31.019 INFO:tasks.workunit.client.0.trial043.stderr:+ expr match 'GNU gdb (CentOS Stream) 16.3-2.el9 2026-02-26T02:43:31.019 INFO:tasks.workunit.client.0.trial043.stderr:Copyright (C) 2024 Free Software Foundation, Inc. 2026-02-26T02:43:31.020 INFO:tasks.workunit.client.0.trial043.stderr:License GPLv3+: GNU GPL version 3 or later 2026-02-26T02:43:31.020 INFO:tasks.workunit.client.0.trial043.stderr:This is free software: you are free to change and redistribute it. 2026-02-26T02:43:31.020 INFO:tasks.workunit.client.0.trial043.stderr:There is NO WARRANTY, to the extent permitted by law. 2026-02-26T02:43:31.020 INFO:tasks.workunit.client.0.trial043.stderr:Type "show copying" and "show warranty" for details. 2026-02-26T02:43:31.020 INFO:tasks.workunit.client.0.trial043.stderr:This GDB was configured as "x86_64-redhat-linux-gnu". 2026-02-26T02:43:31.020 INFO:tasks.workunit.client.0.trial043.stderr:Type "show configuration" for configuration details. 2026-02-26T02:43:31.020 INFO:tasks.workunit.client.0.trial043.stderr:For bug reporting instructions, please see: 2026-02-26T02:43:31.020 INFO:tasks.workunit.client.0.trial043.stderr:. 2026-02-26T02:43:31.020 INFO:tasks.workunit.client.0.trial043.stderr:Find the GDB manual and other documentation resources online at: 2026-02-26T02:43:31.020 INFO:tasks.workunit.client.0.trial043.stderr: . 2026-02-26T02:43:31.021 INFO:tasks.workunit.client.0.trial043.stderr: 2026-02-26T02:43:31.021 INFO:tasks.workunit.client.0.trial043.stderr:For help, type "help". 2026-02-26T02:43:31.021 INFO:tasks.workunit.client.0.trial043.stderr:Type "apropos word" to search for commands related to "word"... 2026-02-26T02:43:31.021 INFO:tasks.workunit.client.0.trial043.stderr:Reading symbols from /usr/bin/ceph-osd... 2026-02-26T02:43:31.021 INFO:tasks.workunit.client.0.trial043.stderr:Reading symbols from .gnu_debugdata for /usr/bin/ceph-osd... 2026-02-26T02:43:31.021 INFO:tasks.workunit.client.0.trial043.stderr:(No debugging symbols found in .gnu_debugdata for /usr/bin/ceph-osd) 2026-02-26T02:43:31.021 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43515] 2026-02-26T02:43:31.021 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43520] 2026-02-26T02:43:31.021 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43522] 2026-02-26T02:43:31.021 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43518] 2026-02-26T02:43:31.021 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43535] 2026-02-26T02:43:31.022 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43759] 2026-02-26T02:43:31.022 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43758] 2026-02-26T02:43:31.022 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43521] 2026-02-26T02:43:31.022 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43768] 2026-02-26T02:43:31.022 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43835] 2026-02-26T02:43:31.022 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43836] 2026-02-26T02:43:31.022 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43756] 2026-02-26T02:43:31.022 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43783] 2026-02-26T02:43:31.022 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43790] 2026-02-26T02:43:31.022 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43534] 2026-02-26T02:43:31.022 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44054] 2026-02-26T02:43:31.023 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45166] 2026-02-26T02:43:31.023 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45759] 2026-02-26T02:43:31.023 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45760] 2026-02-26T02:43:31.023 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45943] 2026-02-26T02:43:31.023 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43797] 2026-02-26T02:43:31.023 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45941] 2026-02-26T02:43:31.023 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43770] 2026-02-26T02:43:31.023 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43778] 2026-02-26T02:43:31.023 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45958] 2026-02-26T02:43:31.023 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45950] 2026-02-26T02:43:31.023 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45968] 2026-02-26T02:43:31.024 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45952] 2026-02-26T02:43:31.024 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45948] 2026-02-26T02:43:31.024 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45969] 2026-02-26T02:43:31.024 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43834] 2026-02-26T02:43:31.024 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45972] 2026-02-26T02:43:31.024 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45984] 2026-02-26T02:43:31.024 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45974] 2026-02-26T02:43:31.024 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43804] 2026-02-26T02:43:31.024 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44055] 2026-02-26T02:43:31.024 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45946] 2026-02-26T02:43:31.024 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45973] 2026-02-26T02:43:31.024 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45966] 2026-02-26T02:43:31.025 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45971] 2026-02-26T02:43:31.025 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45954] 2026-02-26T02:43:31.025 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45980] 2026-02-26T02:43:31.025 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45947] 2026-02-26T02:43:31.025 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45951] 2026-02-26T02:43:31.025 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43813] 2026-02-26T02:43:31.025 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45942] 2026-02-26T02:43:31.025 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45967] 2026-02-26T02:43:31.025 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45944] 2026-02-26T02:43:31.025 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45945] 2026-02-26T02:43:31.026 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44056] 2026-02-26T02:43:31.026 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45975] 2026-02-26T02:43:31.026 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44057] 2026-02-26T02:43:31.026 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45953] 2026-02-26T02:43:31.026 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45956] 2026-02-26T02:43:31.026 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45976] 2026-02-26T02:43:31.026 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45978] 2026-02-26T02:43:31.026 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43820] 2026-02-26T02:43:31.026 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45955] 2026-02-26T02:43:31.026 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45981] 2026-02-26T02:43:31.026 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45957] 2026-02-26T02:43:31.026 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45983] 2026-02-26T02:43:31.027 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45959] 2026-02-26T02:43:31.027 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45982] 2026-02-26T02:43:31.027 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43833] 2026-02-26T02:43:31.027 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43837] 2026-02-26T02:43:31.027 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45949] 2026-02-26T02:43:31.027 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45960] 2026-02-26T02:43:31.027 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45961] 2026-02-26T02:43:31.027 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43838] 2026-02-26T02:43:31.027 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45962] 2026-02-26T02:43:31.027 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45963] 2026-02-26T02:43:31.027 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45964] 2026-02-26T02:43:31.027 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43839] 2026-02-26T02:43:31.028 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45970] 2026-02-26T02:43:31.028 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45165] 2026-02-26T02:43:31.028 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45977] 2026-02-26T02:43:31.028 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45979] 2026-02-26T02:43:31.028 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45965] 2026-02-26T02:43:31.028 INFO:tasks.workunit.client.0.trial043.stderr:[Thread debugging using libthread_db enabled] 2026-02-26T02:43:31.028 INFO:tasks.workunit.client.0.trial043.stderr:Using host libthread_db library "/lib64/libthread_db.so.1". 2026-02-26T02:43:31.028 INFO:tasks.workunit.client.0.trial043.stderr:Core was generated by `/bin/ceph-osd -f --cluster ceph -i 0'\''. 2026-02-26T02:43:31.028 INFO:tasks.workunit.client.0.trial043.stderr:Program terminated with signal SIGABRT, Aborted. 2026-02-26T02:43:31.028 INFO:tasks.workunit.client.0.trial043.stderr:#0 0x00007ff7a308d03c in __pthread_kill_implementation () 2026-02-26T02:43:31.028 INFO:tasks.workunit.client.0.trial043.stderr: from /lib64/libc.so.6 2026-02-26T02:43:31.029 INFO:tasks.workunit.client.0.trial043.stderr:[Current thread is 1 (Thread 0x7ff7a2b34780 (LWP 43515))] 2026-02-26T02:43:31.029 INFO:tasks.workunit.client.0.trial043.stderr:Missing rpms, try: dnf --enablerepo='\''*debug*'\'' install ceph-osd-debuginfo-19.2.3-687.g8d724284.el9.x86_64 gperftools-libs-debuginfo-2.9.1-3.el9.x86_64 libblkid-debuginfo-2.37.4-21.el9.x86_64 fuse-libs-debuginfo-2.9.9-17.el9.x86_64 libaio-debuginfo-0.3.111-13.el9.x86_64 snappy-debuginfo-1.1.8-8.el9.x86_64 lz4-libs-debuginfo-1.9.3-5.el9.x86_64 zlib-debuginfo-1.2.11-41.el9.x86_64 libcurl-debuginfo-7.76.1-40.el9.x86_64 thrift-debuginfo-0.15.0-4.el9.x86_64 libcap-debuginfo-2.48-10.el9.x86_64 glibc-debuginfo-2.34-245.el9.x86_64 openssl-libs-debuginfo-3.5.5-1.el9.x86_64 systemd-libs-debuginfo-252-64.el9.x86_64 libibverbs-debuginfo-61.0-2.el9.x86_64 librdmacm-debuginfo-61.0-2.el9.x86_64 libstdc++-debuginfo-11.5.0-14.el9.x86_64 libgcc-debuginfo-11.5.0-14.el9.x86_64 libunwind-debuginfo-1.6.2-1.el9.x86_64 libnghttp2-debuginfo-1.43.0-6.el9.x86_64 libidn2-debuginfo-2.3.0-7.el9.x86_64 libssh-debuginfo-0.10.4-17.el9.x86_64 libpsl-debuginfo-0.21.1-5.el9.x86_64 krb5-libs-debuginfo-1.21.1-8.el9.x86_64 libcom_err-debuginfo-1.46.5-8.el9.x86_64 openldap-debuginfo-2.6.8-4.el9.x86_64 libbrotli-debuginfo-1.0.9-9.el9.x86_64 libnl3-debuginfo-3.11.0-1.el9.x86_64 libunistring-debuginfo-0.9.10-15.el9.x86_64 keyutils-libs-debuginfo-1.6.3-1.el9.x86_64 libevent-debuginfo-2.1.12-8.el9.x86_64 cyrus-sasl-lib-debuginfo-2.1.27-21.el9.x86_64 libselinux-debuginfo-3.6-3.el9.x86_64 libxcrypt-debuginfo-4.4.18-3.el9.x86_64 pcre2-debuginfo-10.40-6.el9.x86_64 lttng-ust-debuginfo-2.12.0-6.el9.x86_64 numactl-libs-debuginfo-2.0.19-3.el9.x86_64 userspace-rcu-debuginfo-0.12.1-6.el9.x86_64 ceph-base-debuginfo-19.2.3-687.g8d724284.el9.x86_64 qatzip-libs-debuginfo-1.3.1-1.el9.x86_64 qatlib-debuginfo-25.08.0-2.el9.x86_64 liboath-debuginfo-2.6.12-1.el9.x86_64 2026-02-26T02:43:31.029 INFO:tasks.workunit.client.0.trial043.stderr:(gdb) ' '.*terminated.*signal 6.*' 2026-02-26T02:43:31.029 INFO:tasks.workunit.client.0.trial043.stdout:0 2026-02-26T02:43:31.030 INFO:tasks.workunit.client.0.trial043.stderr:+ expr match 'GNU gdb (CentOS Stream) 16.3-2.el9 2026-02-26T02:43:31.030 INFO:tasks.workunit.client.0.trial043.stderr:Copyright (C) 2024 Free Software Foundation, Inc. 2026-02-26T02:43:31.030 INFO:tasks.workunit.client.0.trial043.stderr:License GPLv3+: GNU GPL version 3 or later 2026-02-26T02:43:31.030 INFO:tasks.workunit.client.0.trial043.stderr:This is free software: you are free to change and redistribute it. 2026-02-26T02:43:31.030 INFO:tasks.workunit.client.0.trial043.stderr:There is NO WARRANTY, to the extent permitted by law. 2026-02-26T02:43:31.030 INFO:tasks.workunit.client.0.trial043.stderr:Type "show copying" and "show warranty" for details. 2026-02-26T02:43:31.030 INFO:tasks.workunit.client.0.trial043.stderr:This GDB was configured as "x86_64-redhat-linux-gnu". 2026-02-26T02:43:31.030 INFO:tasks.workunit.client.0.trial043.stderr:Type "show configuration" for configuration details. 2026-02-26T02:43:31.030 INFO:tasks.workunit.client.0.trial043.stderr:For bug reporting instructions, please see: 2026-02-26T02:43:31.031 INFO:tasks.workunit.client.0.trial043.stderr:. 2026-02-26T02:43:31.031 INFO:tasks.workunit.client.0.trial043.stderr:Find the GDB manual and other documentation resources online at: 2026-02-26T02:43:31.031 INFO:tasks.workunit.client.0.trial043.stderr: . 2026-02-26T02:43:31.031 INFO:tasks.workunit.client.0.trial043.stderr: 2026-02-26T02:43:31.031 INFO:tasks.workunit.client.0.trial043.stderr:For help, type "help". 2026-02-26T02:43:31.031 INFO:tasks.workunit.client.0.trial043.stderr:Type "apropos word" to search for commands related to "word"... 2026-02-26T02:43:31.031 INFO:tasks.workunit.client.0.trial043.stderr:Reading symbols from /usr/bin/ceph-osd... 2026-02-26T02:43:31.031 INFO:tasks.workunit.client.0.trial043.stderr:Reading symbols from .gnu_debugdata for /usr/bin/ceph-osd... 2026-02-26T02:43:31.031 INFO:tasks.workunit.client.0.trial043.stderr:(No debugging symbols found in .gnu_debugdata for /usr/bin/ceph-osd) 2026-02-26T02:43:31.031 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43515] 2026-02-26T02:43:31.031 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43520] 2026-02-26T02:43:31.031 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43522] 2026-02-26T02:43:31.032 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43518] 2026-02-26T02:43:31.032 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43535] 2026-02-26T02:43:31.032 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43759] 2026-02-26T02:43:31.032 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43758] 2026-02-26T02:43:31.032 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43521] 2026-02-26T02:43:31.032 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43768] 2026-02-26T02:43:31.032 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43835] 2026-02-26T02:43:31.032 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43836] 2026-02-26T02:43:31.032 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43756] 2026-02-26T02:43:31.032 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43783] 2026-02-26T02:43:31.032 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43790] 2026-02-26T02:43:31.032 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43534] 2026-02-26T02:43:31.032 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44054] 2026-02-26T02:43:31.032 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45166] 2026-02-26T02:43:31.033 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45759] 2026-02-26T02:43:31.033 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45760] 2026-02-26T02:43:31.033 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45943] 2026-02-26T02:43:31.033 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43797] 2026-02-26T02:43:31.033 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45941] 2026-02-26T02:43:31.033 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43770] 2026-02-26T02:43:31.033 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43778] 2026-02-26T02:43:31.033 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45958] 2026-02-26T02:43:31.033 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45950] 2026-02-26T02:43:31.033 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45968] 2026-02-26T02:43:31.033 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45952] 2026-02-26T02:43:31.033 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45948] 2026-02-26T02:43:31.033 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45969] 2026-02-26T02:43:31.034 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43834] 2026-02-26T02:43:31.034 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45972] 2026-02-26T02:43:31.034 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45984] 2026-02-26T02:43:31.034 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45974] 2026-02-26T02:43:31.034 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43804] 2026-02-26T02:43:31.034 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44055] 2026-02-26T02:43:31.034 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45946] 2026-02-26T02:43:31.034 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45973] 2026-02-26T02:43:31.034 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45966] 2026-02-26T02:43:31.034 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45971] 2026-02-26T02:43:31.034 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45954] 2026-02-26T02:43:31.034 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45980] 2026-02-26T02:43:31.034 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45947] 2026-02-26T02:43:31.035 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45951] 2026-02-26T02:43:31.035 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43813] 2026-02-26T02:43:31.035 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45942] 2026-02-26T02:43:31.035 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45967] 2026-02-26T02:43:31.035 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45944] 2026-02-26T02:43:31.035 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45945] 2026-02-26T02:43:31.035 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44056] 2026-02-26T02:43:31.035 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45975] 2026-02-26T02:43:31.035 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 44057] 2026-02-26T02:43:31.035 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45953] 2026-02-26T02:43:31.035 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45956] 2026-02-26T02:43:31.035 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45976] 2026-02-26T02:43:31.036 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45978] 2026-02-26T02:43:31.036 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43820] 2026-02-26T02:43:31.036 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45955] 2026-02-26T02:43:31.036 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45981] 2026-02-26T02:43:31.036 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45957] 2026-02-26T02:43:31.036 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45983] 2026-02-26T02:43:31.036 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45959] 2026-02-26T02:43:31.036 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45982] 2026-02-26T02:43:31.036 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43833] 2026-02-26T02:43:31.036 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43837] 2026-02-26T02:43:31.036 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45949] 2026-02-26T02:43:31.036 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45960] 2026-02-26T02:43:31.036 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45961] 2026-02-26T02:43:31.036 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43838] 2026-02-26T02:43:31.037 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45962] 2026-02-26T02:43:31.037 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45963] 2026-02-26T02:43:31.037 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45964] 2026-02-26T02:43:31.037 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 43839] 2026-02-26T02:43:31.037 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45970] 2026-02-26T02:43:31.037 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45165] 2026-02-26T02:43:31.037 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45977] 2026-02-26T02:43:31.037 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45979] 2026-02-26T02:43:31.037 INFO:tasks.workunit.client.0.trial043.stderr:[New LWP 45965] 2026-02-26T02:43:31.037 INFO:tasks.workunit.client.0.trial043.stderr:[Thread debugging using libthread_db enabled] 2026-02-26T02:43:31.037 INFO:tasks.workunit.client.0.trial043.stderr:Using host libthread_db library "/lib64/libthread_db.so.1". 2026-02-26T02:43:31.037 INFO:tasks.workunit.client.0.trial043.stderr:Core was generated by `/bin/ceph-osd -f --cluster ceph -i 0'\''. 2026-02-26T02:43:31.037 INFO:tasks.workunit.client.0.trial043.stderr:Program terminated with signal SIGABRT, Aborted. 2026-02-26T02:43:31.037 INFO:tasks.workunit.client.0.trial043.stderr:#0 0x00007ff7a308d03c in __pthread_kill_implementation () 2026-02-26T02:43:31.038 INFO:tasks.workunit.client.0.trial043.stderr: from /lib64/libc.so.6 2026-02-26T02:43:31.038 INFO:tasks.workunit.client.0.trial043.stderr:[Current thread is 1 (Thread 0x7ff7a2b34780 (LWP 43515))] 2026-02-26T02:43:31.038 INFO:tasks.workunit.client.0.trial043.stderr:Missing rpms, try: dnf --enablerepo='\''*debug*'\'' install ceph-osd-debuginfo-19.2.3-687.g8d724284.el9.x86_64 gperftools-libs-debuginfo-2.9.1-3.el9.x86_64 libblkid-debuginfo-2.37.4-21.el9.x86_64 fuse-libs-debuginfo-2.9.9-17.el9.x86_64 libaio-debuginfo-0.3.111-13.el9.x86_64 snappy-debuginfo-1.1.8-8.el9.x86_64 lz4-libs-debuginfo-1.9.3-5.el9.x86_64 zlib-debuginfo-1.2.11-41.el9.x86_64 libcurl-debuginfo-7.76.1-40.el9.x86_64 thrift-debuginfo-0.15.0-4.el9.x86_64 libcap-debuginfo-2.48-10.el9.x86_64 glibc-debuginfo-2.34-245.el9.x86_64 openssl-libs-debuginfo-3.5.5-1.el9.x86_64 systemd-libs-debuginfo-252-64.el9.x86_64 libibverbs-debuginfo-61.0-2.el9.x86_64 librdmacm-debuginfo-61.0-2.el9.x86_64 libstdc++-debuginfo-11.5.0-14.el9.x86_64 libgcc-debuginfo-11.5.0-14.el9.x86_64 libunwind-debuginfo-1.6.2-1.el9.x86_64 libnghttp2-debuginfo-1.43.0-6.el9.x86_64 libidn2-debuginfo-2.3.0-7.el9.x86_64 libssh-debuginfo-0.10.4-17.el9.x86_64 libpsl-debuginfo-0.21.1-5.el9.x86_64 krb5-libs-debuginfo-1.21.1-8.el9.x86_64 libcom_err-debuginfo-1.46.5-8.el9.x86_64 openldap-debuginfo-2.6.8-4.el9.x86_64 libbrotli-debuginfo-1.0.9-9.el9.x86_64 libnl3-debuginfo-3.11.0-1.el9.x86_64 libunistring-debuginfo-0.9.10-15.el9.x86_64 keyutils-libs-debuginfo-1.6.3-1.el9.x86_64 libevent-debuginfo-2.1.12-8.el9.x86_64 cyrus-sasl-lib-debuginfo-2.1.27-21.el9.x86_64 libselinux-debuginfo-3.6-3.el9.x86_64 libxcrypt-debuginfo-4.4.18-3.el9.x86_64 pcre2-debuginfo-10.40-6.el9.x86_64 lttng-ust-debuginfo-2.12.0-6.el9.x86_64 numactl-libs-debuginfo-2.0.19-3.el9.x86_64 userspace-rcu-debuginfo-0.12.1-6.el9.x86_64 ceph-base-debuginfo-19.2.3-687.g8d724284.el9.x86_64 qatzip-libs-debuginfo-1.3.1-1.el9.x86_64 qatlib-debuginfo-25.08.0-2.el9.x86_64 liboath-debuginfo-2.6.12-1.el9.x86_64 2026-02-26T02:43:31.038 INFO:tasks.workunit.client.0.trial043.stderr:(gdb) ' '.*terminated.*signal SIGABRT.*' 2026-02-26T02:43:31.038 INFO:tasks.workunit.client.0.trial043.stdout:4256 2026-02-26T02:43:31.039 INFO:tasks.workunit.client.0.trial043.stderr:+ sudo rm /home/ubuntu/cephtest/archive/coredump/1772073805.43515.core 2026-02-26T02:43:31.039 INFO:tasks.workunit.client.0.trial043.stderr:+ sudo chown -R ceph:ceph /var/lib/ceph/crash 2026-02-26T02:43:31.039 INFO:tasks.workunit.client.0.trial043.stderr:+ sudo systemctl restart ceph-crash 2026-02-26T02:43:31.059 INFO:tasks.workunit.client.0.trial043.stderr:+ sleep 30 2026-02-26T02:43:33.258 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~5s 2026-02-26T02:43:33.258 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~5s 2026-02-26T02:43:33.258 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~5s 2026-02-26T02:43:38.460 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~10s 2026-02-26T02:43:38.460 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~10s 2026-02-26T02:43:38.460 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~10s 2026-02-26T02:43:43.662 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~16s 2026-02-26T02:43:43.662 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~16s 2026-02-26T02:43:43.663 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~16s 2026-02-26T02:43:48.865 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~21s 2026-02-26T02:43:48.865 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~21s 2026-02-26T02:43:48.865 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~21s 2026-02-26T02:43:54.068 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~26s 2026-02-26T02:43:54.068 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~26s 2026-02-26T02:43:54.068 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~26s 2026-02-26T02:43:59.269 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~31s 2026-02-26T02:43:59.270 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~31s 2026-02-26T02:43:59.270 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~31s 2026-02-26T02:44:01.061 INFO:tasks.workunit.client.0.trial043.stderr:++ ceph crash ls 2026-02-26T02:44:01.061 INFO:tasks.workunit.client.0.trial043.stderr:++ wc -l 2026-02-26T02:44:01.253 INFO:tasks.workunit.client.0.trial043.stderr:+ '[' 4 = 4 ']' 2026-02-26T02:44:01.254 INFO:tasks.workunit.client.0.trial043.stderr:++ sudo find /var/lib/ceph/crash/posted/ -name meta 2026-02-26T02:44:01.254 INFO:tasks.workunit.client.0.trial043.stderr:++ wc -l 2026-02-26T02:44:01.264 INFO:tasks.workunit.client.0.trial043.stderr:+ '[' 3 = 3 ']' 2026-02-26T02:44:01.264 INFO:tasks.workunit.client.0.trial043.stderr:+ ceph health detail 2026-02-26T02:44:01.265 INFO:tasks.workunit.client.0.trial043.stderr:+ grep RECENT_CRASH 2026-02-26T02:44:01.510 INFO:tasks.workunit.client.0.trial043.stdout:[WRN] RECENT_CRASH: 3 daemons have recently crashed 2026-02-26T02:44:01.511 INFO:tasks.workunit.client.0.trial043.stderr:+ ceph crash archive-all 2026-02-26T02:44:01.709 INFO:tasks.workunit.client.0.trial043.stderr:+ sleep 30 2026-02-26T02:44:04.471 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~36s 2026-02-26T02:44:04.472 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~36s 2026-02-26T02:44:04.472 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~36s 2026-02-26T02:44:09.674 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~42s 2026-02-26T02:44:09.675 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~42s 2026-02-26T02:44:09.675 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~42s 2026-02-26T02:44:14.877 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~47s 2026-02-26T02:44:14.877 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~47s 2026-02-26T02:44:14.877 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~47s 2026-02-26T02:44:20.079 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~52s 2026-02-26T02:44:20.079 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~52s 2026-02-26T02:44:20.079 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~52s 2026-02-26T02:44:25.281 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~57s 2026-02-26T02:44:25.282 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~57s 2026-02-26T02:44:25.282 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~57s 2026-02-26T02:44:30.484 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~62s 2026-02-26T02:44:30.484 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~62s 2026-02-26T02:44:30.484 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~62s 2026-02-26T02:44:31.710 INFO:tasks.workunit.client.0.trial043.stderr:+ ceph health detail 2026-02-26T02:44:31.711 INFO:tasks.workunit.client.0.trial043.stderr:+ grep -c RECENT_CRASH 2026-02-26T02:44:31.711 INFO:tasks.workunit.client.0.trial043.stderr:+ grep 0 2026-02-26T02:44:31.965 INFO:tasks.workunit.client.0.trial043.stdout:0 2026-02-26T02:44:31.965 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-26T02:44:31.966 DEBUG:teuthology.orchestra.run.trial043:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-26T02:44:31.991 INFO:tasks.workunit:Stopping ['rados/test_crash.sh'] on client.0... 2026-02-26T02:44:31.992 DEBUG:teuthology.orchestra.run.trial043:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2026-02-26T02:44:32.329 DEBUG:teuthology.parallel:result is None 2026-02-26T02:44:32.330 DEBUG:teuthology.orchestra.run.trial043:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2026-02-26T02:44:32.356 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2026-02-26T02:44:32.356 DEBUG:teuthology.orchestra.run.trial043:> rmdir -- /home/ubuntu/cephtest/mnt.0 2026-02-26T02:44:32.413 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2026-02-26T02:44:32.413 INFO:teuthology.run_tasks:Running task ceph.restart... 2026-02-26T02:44:32.438 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-26T02:44:32.438 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T02:44:32.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T02:44:32.439 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/orchestra/run.py", line 485, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial043 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0' 2026-02-26T02:44:32.443 INFO:tasks.ceph.osd.0:Stopped 2026-02-26T02:44:32.443 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 0 2026-02-26T02:44:32.611 INFO:tasks.ceph.mon.a.trial043.stderr:2026-02-26T02:44:32.608+0000 7f6a74680640 -1 mon.a@0(leader).osd e16 definitely_dead 0 2026-02-26T02:44:33.055 INFO:teuthology.orchestra.run.trial043.stderr:marked down osd.0. 2026-02-26T02:44:33.067 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-26T02:44:33.067 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2026-02-26T02:44:33.069 INFO:tasks.ceph.osd.0:Started 2026-02-26T02:44:33.069 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-26T02:44:33.069 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T02:44:33.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T02:44:33.069 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/orchestra/run.py", line 485, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial043 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1' 2026-02-26T02:44:33.070 INFO:tasks.ceph.osd.1:Stopped 2026-02-26T02:44:33.070 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 1 2026-02-26T02:44:33.230 INFO:tasks.ceph.mon.a.trial043.stderr:2026-02-26T02:44:33.227+0000 7f6a74680640 -1 mon.a@0(leader).osd e17 definitely_dead 0 2026-02-26T02:44:34.060 INFO:teuthology.orchestra.run.trial043.stderr:marked down osd.1. 2026-02-26T02:44:34.071 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-26T02:44:34.071 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2026-02-26T02:44:34.072 INFO:tasks.ceph.osd.1:Started 2026-02-26T02:44:34.072 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-26T02:44:34.072 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T02:44:34.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T02:44:34.072 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/orchestra/run.py", line 485, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial043 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2' 2026-02-26T02:44:34.073 INFO:tasks.ceph.osd.2:Stopped 2026-02-26T02:44:34.073 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 2 2026-02-26T02:44:34.228 INFO:tasks.ceph.mon.a.trial043.stderr:2026-02-26T02:44:34.225+0000 7f6a74680640 -1 mon.a@0(leader).osd e18 definitely_dead 0 2026-02-26T02:44:34.954 INFO:tasks.ceph.osd.0.trial043.stderr:2026-02-26T02:44:34.951+0000 7f449b128780 -1 Falling back to public interface 2026-02-26T02:44:35.064 INFO:teuthology.orchestra.run.trial043.stderr:marked down osd.2. 2026-02-26T02:44:35.075 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-26T02:44:35.075 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2026-02-26T02:44:35.077 INFO:tasks.ceph.osd.2:Started 2026-02-26T02:44:35.078 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-26T02:44:35.078 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-26T02:44:35.078 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-26T02:44:35.258 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:44:35.265 INFO:teuthology.orchestra.run.trial043.stdout:{"epoch":4,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6800","nonce":3619129573}]},"active_addr":"10.20.193.43:6800/3619129573","active_change":"2026-02-26T02:42:42.267817+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number. if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.43:0","nonce":3642507123}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.43:0","nonce":2134978891}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.43:0","nonce":3371346229}]}]} 2026-02-26T02:44:35.267 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-26T02:44:35.267 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-26T02:44:35.267 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T02:44:35.426 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:44:35.426 INFO:teuthology.orchestra.run.trial043.stdout:{"epoch":19,"fsid":"6dea32ae-afd8-4e41-a06e-2ec4e4797235","created":"2026-02-26T02:42:40.023737+0000","modified":"2026-02-26T02:44:35.059184+0000","last_up_change":"2026-02-26T02:44:35.059184+0000","last_in_change":"2026-02-26T02:42:40.696050+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-26T02:42:54.301050+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"3383d670-1454-4abb-8847-9dab847e0a5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6805","nonce":1923240457}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6807","nonce":1923240457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6812","nonce":1923240457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6809","nonce":1923240457}]},"public_addr":"10.20.193.43:6805/1923240457","cluster_addr":"10.20.193.43:6807/1923240457","heartbeat_back_addr":"10.20.193.43:6812/1923240457","heartbeat_front_addr":"10.20.193.43:6809/1923240457","state":["exists"]},{"osd":1,"uuid":"729a8cd6-7014-40b4-9e02-940e867460f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6806","nonce":4263071973}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6808","nonce":4263071973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6811","nonce":4263071973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6810","nonce":4263071973}]},"public_addr":"10.20.193.43:6806/4263071973","cluster_addr":"10.20.193.43:6808/4263071973","heartbeat_back_addr":"10.20.193.43:6811/4263071973","heartbeat_front_addr":"10.20.193.43:6810/4263071973","state":["exists"]},{"osd":2,"uuid":"b1591962-eecc-4a4f-83d1-040bee63ba29","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":19,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6801","nonce":1886150811}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6802","nonce":1886150811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6804","nonce":1886150811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6803","nonce":1886150811}]},"public_addr":"10.20.193.43:6801/1886150811","cluster_addr":"10.20.193.43:6802/1886150811","heartbeat_back_addr":"10.20.193.43:6804/1886150811","heartbeat_front_addr":"10.20.193.43:6803/1886150811","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2026-02-26T02:44:33.052845+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T02:42:48.220681+0000","dead_epoch":0},{"osd":1,"down_stamp":"2026-02-26T02:44:34.056568+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T02:42:48.293491+0000","dead_epoch":0},{"osd":2,"down_stamp":"2026-02-26T02:44:35.059184+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T02:42:48.146728+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T02:44:35.978 INFO:tasks.ceph.osd.1.trial043.stderr:2026-02-26T02:44:35.975+0000 7f8f1dd3f780 -1 Falling back to public interface 2026-02-26T02:44:35.988 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2026-02-26T02:44:35.988 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2026-02-26T02:44:35.988 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2026-02-26T02:44:36.960 INFO:tasks.ceph.osd.2.trial043.stderr:2026-02-26T02:44:36.957+0000 7f478801c780 -1 Falling back to public interface 2026-02-26T02:44:38.436 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T02:44:38.592 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:44:38.592 INFO:teuthology.orchestra.run.trial043.stdout:{"epoch":19,"fsid":"6dea32ae-afd8-4e41-a06e-2ec4e4797235","created":"2026-02-26T02:42:40.023737+0000","modified":"2026-02-26T02:44:35.059184+0000","last_up_change":"2026-02-26T02:44:35.059184+0000","last_in_change":"2026-02-26T02:42:40.696050+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-26T02:42:54.301050+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"3383d670-1454-4abb-8847-9dab847e0a5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6805","nonce":1923240457}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6807","nonce":1923240457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6812","nonce":1923240457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6809","nonce":1923240457}]},"public_addr":"10.20.193.43:6805/1923240457","cluster_addr":"10.20.193.43:6807/1923240457","heartbeat_back_addr":"10.20.193.43:6812/1923240457","heartbeat_front_addr":"10.20.193.43:6809/1923240457","state":["exists"]},{"osd":1,"uuid":"729a8cd6-7014-40b4-9e02-940e867460f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6806","nonce":4263071973}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6808","nonce":4263071973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6811","nonce":4263071973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6810","nonce":4263071973}]},"public_addr":"10.20.193.43:6806/4263071973","cluster_addr":"10.20.193.43:6808/4263071973","heartbeat_back_addr":"10.20.193.43:6811/4263071973","heartbeat_front_addr":"10.20.193.43:6810/4263071973","state":["exists"]},{"osd":2,"uuid":"b1591962-eecc-4a4f-83d1-040bee63ba29","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":19,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6801","nonce":1886150811}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6802","nonce":1886150811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6804","nonce":1886150811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6803","nonce":1886150811}]},"public_addr":"10.20.193.43:6801/1886150811","cluster_addr":"10.20.193.43:6802/1886150811","heartbeat_back_addr":"10.20.193.43:6804/1886150811","heartbeat_front_addr":"10.20.193.43:6803/1886150811","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2026-02-26T02:44:33.052845+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T02:42:48.220681+0000","dead_epoch":0},{"osd":1,"down_stamp":"2026-02-26T02:44:34.056568+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T02:42:48.293491+0000","dead_epoch":0},{"osd":2,"down_stamp":"2026-02-26T02:44:35.059184+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T02:42:48.146728+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T02:44:39.444 INFO:tasks.ceph.osd.0.trial043.stderr:2026-02-26T02:44:39.442+0000 7f449b128780 -1 osd.0 16 log_to_monitors true 2026-02-26T02:44:40.072 INFO:tasks.ceph.osd.0.trial043.stderr:2026-02-26T02:44:40.070+0000 7f448eec6640 -1 osd.0 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-26T02:44:40.491 INFO:tasks.ceph.osd.1.trial043.stderr:2026-02-26T02:44:40.488+0000 7f8f1dd3f780 -1 osd.1 16 log_to_monitors true 2026-02-26T02:44:41.345 INFO:tasks.ceph.osd.1.trial043.stderr:2026-02-26T02:44:41.343+0000 7f8f11adf640 -1 osd.1 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-26T02:44:41.449 INFO:tasks.ceph.osd.2.trial043.stderr:2026-02-26T02:44:41.447+0000 7f478801c780 -1 osd.2 16 log_to_monitors true 2026-02-26T02:44:41.602 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T02:44:41.757 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:44:41.757 INFO:teuthology.orchestra.run.trial043.stdout:{"epoch":21,"fsid":"6dea32ae-afd8-4e41-a06e-2ec4e4797235","created":"2026-02-26T02:42:40.023737+0000","modified":"2026-02-26T02:44:41.066781+0000","last_up_change":"2026-02-26T02:44:41.066781+0000","last_in_change":"2026-02-26T02:42:40.696050+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-26T02:42:54.301050+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"error_message":"pool 1 has some PGs where all OSDs are with primary_affinity 0 (at least pg 1.3), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"3383d670-1454-4abb-8847-9dab847e0a5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":16,"up_from":21,"up_thru":10,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6801","nonce":2557732258}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6802","nonce":2557732258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6804","nonce":2557732258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6803","nonce":2557732258}]},"public_addr":"10.20.193.43:6801/2557732258","cluster_addr":"10.20.193.43:6802/2557732258","heartbeat_back_addr":"10.20.193.43:6804/2557732258","heartbeat_front_addr":"10.20.193.43:6803/2557732258","state":["exists","up"]},{"osd":1,"uuid":"729a8cd6-7014-40b4-9e02-940e867460f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6806","nonce":4263071973}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6808","nonce":4263071973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6811","nonce":4263071973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6810","nonce":4263071973}]},"public_addr":"10.20.193.43:6806/4263071973","cluster_addr":"10.20.193.43:6808/4263071973","heartbeat_back_addr":"10.20.193.43:6811/4263071973","heartbeat_front_addr":"10.20.193.43:6810/4263071973","state":["exists"]},{"osd":2,"uuid":"b1591962-eecc-4a4f-83d1-040bee63ba29","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":19,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6801","nonce":1886150811}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6802","nonce":1886150811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6804","nonce":1886150811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6803","nonce":1886150811}]},"public_addr":"10.20.193.43:6801/1886150811","cluster_addr":"10.20.193.43:6802/1886150811","heartbeat_back_addr":"10.20.193.43:6804/1886150811","heartbeat_front_addr":"10.20.193.43:6803/1886150811","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2026-02-26T02:44:33.052845+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T02:42:48.220681+0000","dead_epoch":0},{"osd":1,"down_stamp":"2026-02-26T02:44:34.056568+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T02:42:48.293491+0000","dead_epoch":0},{"osd":2,"down_stamp":"2026-02-26T02:44:35.059184+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T02:42:48.146728+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T02:44:42.079 INFO:tasks.ceph.osd.2.trial043.stderr:2026-02-26T02:44:42.077+0000 7f477f78f640 -1 osd.2 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-26T02:44:44.765 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T02:44:50.924 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:44:50.924 INFO:teuthology.orchestra.run.trial043.stdout:{"epoch":24,"fsid":"6dea32ae-afd8-4e41-a06e-2ec4e4797235","created":"2026-02-26T02:42:40.023737+0000","modified":"2026-02-26T02:44:44.075883+0000","last_up_change":"2026-02-26T02:44:43.073961+0000","last_in_change":"2026-02-26T02:42:40.696050+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-26T02:42:54.301050+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3383d670-1454-4abb-8847-9dab847e0a5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":16,"up_from":21,"up_thru":22,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6801","nonce":2557732258}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6802","nonce":2557732258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6804","nonce":2557732258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6803","nonce":2557732258}]},"public_addr":"10.20.193.43:6801/2557732258","cluster_addr":"10.20.193.43:6802/2557732258","heartbeat_back_addr":"10.20.193.43:6804/2557732258","heartbeat_front_addr":"10.20.193.43:6803/2557732258","state":["exists","up"]},{"osd":1,"uuid":"729a8cd6-7014-40b4-9e02-940e867460f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":22,"up_thru":23,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6805","nonce":2010462298}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6806","nonce":2010462298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6808","nonce":2010462298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6807","nonce":2010462298}]},"public_addr":"10.20.193.43:6805/2010462298","cluster_addr":"10.20.193.43:6806/2010462298","heartbeat_back_addr":"10.20.193.43:6808/2010462298","heartbeat_front_addr":"10.20.193.43:6807/2010462298","state":["exists","up"]},{"osd":2,"uuid":"b1591962-eecc-4a4f-83d1-040bee63ba29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":16,"up_from":23,"up_thru":23,"down_at":19,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6809","nonce":2224833790}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6810","nonce":2224833790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6812","nonce":2224833790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6811","nonce":2224833790}]},"public_addr":"10.20.193.43:6809/2224833790","cluster_addr":"10.20.193.43:6810/2224833790","heartbeat_back_addr":"10.20.193.43:6812/2224833790","heartbeat_front_addr":"10.20.193.43:6811/2224833790","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2026-02-26T02:44:33.052845+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T02:42:48.220681+0000","dead_epoch":0},{"osd":1,"down_stamp":"2026-02-26T02:44:34.056568+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T02:42:48.293491+0000","dead_epoch":0},{"osd":2,"down_stamp":"2026-02-26T02:44:35.059184+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T02:42:48.146728+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T02:44:50.932 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-26T02:44:50.933 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T02:44:51.087 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:44:51.087 INFO:teuthology.orchestra.run.trial043.stdout:{"epoch":24,"fsid":"6dea32ae-afd8-4e41-a06e-2ec4e4797235","created":"2026-02-26T02:42:40.023737+0000","modified":"2026-02-26T02:44:44.075883+0000","last_up_change":"2026-02-26T02:44:43.073961+0000","last_in_change":"2026-02-26T02:42:40.696050+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-26T02:42:54.301050+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3383d670-1454-4abb-8847-9dab847e0a5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":16,"up_from":21,"up_thru":22,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6801","nonce":2557732258}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6802","nonce":2557732258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6804","nonce":2557732258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6803","nonce":2557732258}]},"public_addr":"10.20.193.43:6801/2557732258","cluster_addr":"10.20.193.43:6802/2557732258","heartbeat_back_addr":"10.20.193.43:6804/2557732258","heartbeat_front_addr":"10.20.193.43:6803/2557732258","state":["exists","up"]},{"osd":1,"uuid":"729a8cd6-7014-40b4-9e02-940e867460f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":22,"up_thru":23,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6805","nonce":2010462298}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6806","nonce":2010462298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6808","nonce":2010462298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6807","nonce":2010462298}]},"public_addr":"10.20.193.43:6805/2010462298","cluster_addr":"10.20.193.43:6806/2010462298","heartbeat_back_addr":"10.20.193.43:6808/2010462298","heartbeat_front_addr":"10.20.193.43:6807/2010462298","state":["exists","up"]},{"osd":2,"uuid":"b1591962-eecc-4a4f-83d1-040bee63ba29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":16,"up_from":23,"up_thru":23,"down_at":19,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6809","nonce":2224833790}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6810","nonce":2224833790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6812","nonce":2224833790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6811","nonce":2224833790}]},"public_addr":"10.20.193.43:6809/2224833790","cluster_addr":"10.20.193.43:6810/2224833790","heartbeat_back_addr":"10.20.193.43:6812/2224833790","heartbeat_front_addr":"10.20.193.43:6811/2224833790","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2026-02-26T02:44:33.052845+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T02:42:48.220681+0000","dead_epoch":0},{"osd":1,"down_stamp":"2026-02-26T02:44:34.056568+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T02:42:48.293491+0000","dead_epoch":0},{"osd":2,"down_stamp":"2026-02-26T02:44:35.059184+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T02:42:48.146728+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T02:44:51.096 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-26T02:44:51.096 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-26T02:44:51.096 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-26T02:44:51.181 INFO:teuthology.orchestra.run.trial043.stdout:94489280516 2026-02-26T02:44:51.181 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-26T02:44:51.182 INFO:teuthology.orchestra.run.trial043.stdout:98784247811 2026-02-26T02:44:51.182 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-26T02:44:51.182 INFO:teuthology.orchestra.run.trial043.stdout:90194313221 2026-02-26T02:44:51.182 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-26T02:44:51.382 INFO:teuthology.orchestra.run.trial043.stdout:94489280514 2026-02-26T02:44:51.386 INFO:teuthology.orchestra.run.trial043.stdout:98784247810 2026-02-26T02:44:51.393 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280514 for osd.1 2026-02-26T02:44:51.396 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247811 got 98784247810 for osd.2 2026-02-26T02:44:51.421 INFO:teuthology.orchestra.run.trial043.stdout:90194313219 2026-02-26T02:44:51.430 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313221 got 90194313219 for osd.0 2026-02-26T02:44:52.393 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-26T02:44:52.396 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-26T02:44:52.431 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-26T02:44:52.555 INFO:teuthology.orchestra.run.trial043.stdout:94489280516 2026-02-26T02:44:52.565 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.1 2026-02-26T02:44:52.565 DEBUG:teuthology.parallel:result is None 2026-02-26T02:44:52.592 INFO:teuthology.orchestra.run.trial043.stdout:90194313221 2026-02-26T02:44:52.596 INFO:teuthology.orchestra.run.trial043.stdout:98784247811 2026-02-26T02:44:52.602 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313221 got 90194313221 for osd.0 2026-02-26T02:44:52.602 DEBUG:teuthology.parallel:result is None 2026-02-26T02:44:52.606 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247811 got 98784247811 for osd.2 2026-02-26T02:44:52.606 DEBUG:teuthology.parallel:result is None 2026-02-26T02:44:52.607 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-26T02:44:52.607 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T02:44:52.799 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:44:52.799 INFO:teuthology.orchestra.run.trial043.stderr:dumped all 2026-02-26T02:44:52.807 INFO:teuthology.orchestra.run.trial043.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2026-02-26T02:44:51.654362+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":87208,"kb_used_data":608,"kb_used_omap":12,"kb_used_meta":86515,"kb_avail":283028312,"statfs":{"total":289910292480,"available":289820991488,"internally_reserved":0,"allocated":622592,"data_stored":169478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13295,"internal_metadata":88591377},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.579687"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":38,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.080071+0000","last_change":"2026-02-26T02:44:44.080071+0000","last_active":"2026-02-26T02:44:44.080071+0000","last_peered":"2026-02-26T02:44:44.080071+0000","last_clean":"2026-02-26T02:44:44.080071+0000","last_became_active":"2026-02-26T02:44:44.079848+0000","last_became_peered":"2026-02-26T02:44:44.079848+0000","last_unstale":"2026-02-26T02:44:44.080071+0000","last_undegraded":"2026-02-26T02:44:44.080071+0000","last_fullsized":"2026-02-26T02:44:44.080071+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T06:53:14.722053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047321700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.078948+0000","last_change":"2026-02-26T02:44:43.077939+0000","last_active":"2026-02-26T02:44:44.078948+0000","last_peered":"2026-02-26T02:44:44.078948+0000","last_clean":"2026-02-26T02:44:44.078948+0000","last_became_active":"2026-02-26T02:44:43.077596+0000","last_became_peered":"2026-02-26T02:44:43.077596+0000","last_unstale":"2026-02-26T02:44:44.078948+0000","last_undegraded":"2026-02-26T02:44:44.078948+0000","last_fullsized":"2026-02-26T02:44:44.078948+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T08:45:22.327951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055368699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.080409+0000","last_change":"2026-02-26T02:44:44.080409+0000","last_active":"2026-02-26T02:44:44.080409+0000","last_peered":"2026-02-26T02:44:44.080409+0000","last_clean":"2026-02-26T02:44:44.080409+0000","last_became_active":"2026-02-26T02:44:44.080002+0000","last_became_peered":"2026-02-26T02:44:44.080002+0000","last_unstale":"2026-02-26T02:44:44.080409+0000","last_undegraded":"2026-02-26T02:44:44.080409+0000","last_fullsized":"2026-02-26T02:44:44.080409+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T12:15:28.872976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.078909+0000","last_change":"2026-02-26T02:44:43.078220+0000","last_active":"2026-02-26T02:44:44.078909+0000","last_peered":"2026-02-26T02:44:44.078909+0000","last_clean":"2026-02-26T02:44:44.078909+0000","last_became_active":"2026-02-26T02:44:43.077809+0000","last_became_peered":"2026-02-26T02:44:43.077809+0000","last_unstale":"2026-02-26T02:44:44.078909+0000","last_undegraded":"2026-02-26T02:44:44.078909+0000","last_fullsized":"2026-02-26T02:44:44.078909+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T13:40:18.032882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044036699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.080508+0000","last_change":"2026-02-26T02:44:44.080508+0000","last_active":"2026-02-26T02:44:44.080508+0000","last_peered":"2026-02-26T02:44:44.080508+0000","last_clean":"2026-02-26T02:44:44.080508+0000","last_became_active":"2026-02-26T02:44:44.080233+0000","last_became_peered":"2026-02-26T02:44:44.080233+0000","last_unstale":"2026-02-26T02:44:44.080508+0000","last_undegraded":"2026-02-26T02:44:44.080508+0000","last_fullsized":"2026-02-26T02:44:44.080508+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T04:07:41.024651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035731199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"15'2","reported_seq":42,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.077881+0000","last_change":"2026-02-26T02:44:43.078398+0000","last_active":"2026-02-26T02:44:44.077881+0000","last_peered":"2026-02-26T02:44:44.077881+0000","last_clean":"2026-02-26T02:44:44.077881+0000","last_became_active":"2026-02-26T02:44:43.077765+0000","last_became_peered":"2026-02-26T02:44:43.077765+0000","last_unstale":"2026-02-26T02:44:44.077881+0000","last_undegraded":"2026-02-26T02:44:44.077881+0000","last_fullsized":"2026-02-26T02:44:44.077881+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T05:33:48.482483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027048099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.079977+0000","last_change":"2026-02-26T02:44:44.079977+0000","last_active":"2026-02-26T02:44:44.079977+0000","last_peered":"2026-02-26T02:44:44.079977+0000","last_clean":"2026-02-26T02:44:44.079977+0000","last_became_active":"2026-02-26T02:44:44.079664+0000","last_became_peered":"2026-02-26T02:44:44.079664+0000","last_unstale":"2026-02-26T02:44:44.079977+0000","last_undegraded":"2026-02-26T02:44:44.079977+0000","last_fullsized":"2026-02-26T02:44:44.079977+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T08:02:36.823659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.078837+0000","last_change":"2026-02-26T02:44:43.078034+0000","last_active":"2026-02-26T02:44:44.078837+0000","last_peered":"2026-02-26T02:44:44.078837+0000","last_clean":"2026-02-26T02:44:44.078837+0000","last_became_active":"2026-02-26T02:44:43.077702+0000","last_became_peered":"2026-02-26T02:44:43.077702+0000","last_unstale":"2026-02-26T02:44:44.078837+0000","last_undegraded":"2026-02-26T02:44:44.078837+0000","last_fullsized":"2026-02-26T02:44:44.078837+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T10:01:12.136568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036294300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31840,"kb_used_data":204,"kb_used_omap":4,"kb_used_meta":31611,"kb_avail":94340000,"statfs":{"total":96636764160,"available":96604160000,"internally_reserved":0,"allocated":208896,"data_stored":56499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4430,"internal_metadata":32370354},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247811,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":200,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":204800,"data_stored":56480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4431,"internal_metadata":28110513},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313221,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":204,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":208896,"data_stored":56499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4434,"internal_metadata":28110510},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T02:44:52.807 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T02:44:52.961 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:44:52.961 INFO:teuthology.orchestra.run.trial043.stderr:dumped all 2026-02-26T02:44:52.969 INFO:teuthology.orchestra.run.trial043.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2026-02-26T02:44:51.654362+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":87208,"kb_used_data":608,"kb_used_omap":12,"kb_used_meta":86515,"kb_avail":283028312,"statfs":{"total":289910292480,"available":289820991488,"internally_reserved":0,"allocated":622592,"data_stored":169478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13295,"internal_metadata":88591377},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.579687"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":38,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.080071+0000","last_change":"2026-02-26T02:44:44.080071+0000","last_active":"2026-02-26T02:44:44.080071+0000","last_peered":"2026-02-26T02:44:44.080071+0000","last_clean":"2026-02-26T02:44:44.080071+0000","last_became_active":"2026-02-26T02:44:44.079848+0000","last_became_peered":"2026-02-26T02:44:44.079848+0000","last_unstale":"2026-02-26T02:44:44.080071+0000","last_undegraded":"2026-02-26T02:44:44.080071+0000","last_fullsized":"2026-02-26T02:44:44.080071+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T06:53:14.722053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047321700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.078948+0000","last_change":"2026-02-26T02:44:43.077939+0000","last_active":"2026-02-26T02:44:44.078948+0000","last_peered":"2026-02-26T02:44:44.078948+0000","last_clean":"2026-02-26T02:44:44.078948+0000","last_became_active":"2026-02-26T02:44:43.077596+0000","last_became_peered":"2026-02-26T02:44:43.077596+0000","last_unstale":"2026-02-26T02:44:44.078948+0000","last_undegraded":"2026-02-26T02:44:44.078948+0000","last_fullsized":"2026-02-26T02:44:44.078948+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T08:45:22.327951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055368699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.080409+0000","last_change":"2026-02-26T02:44:44.080409+0000","last_active":"2026-02-26T02:44:44.080409+0000","last_peered":"2026-02-26T02:44:44.080409+0000","last_clean":"2026-02-26T02:44:44.080409+0000","last_became_active":"2026-02-26T02:44:44.080002+0000","last_became_peered":"2026-02-26T02:44:44.080002+0000","last_unstale":"2026-02-26T02:44:44.080409+0000","last_undegraded":"2026-02-26T02:44:44.080409+0000","last_fullsized":"2026-02-26T02:44:44.080409+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T12:15:28.872976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.078909+0000","last_change":"2026-02-26T02:44:43.078220+0000","last_active":"2026-02-26T02:44:44.078909+0000","last_peered":"2026-02-26T02:44:44.078909+0000","last_clean":"2026-02-26T02:44:44.078909+0000","last_became_active":"2026-02-26T02:44:43.077809+0000","last_became_peered":"2026-02-26T02:44:43.077809+0000","last_unstale":"2026-02-26T02:44:44.078909+0000","last_undegraded":"2026-02-26T02:44:44.078909+0000","last_fullsized":"2026-02-26T02:44:44.078909+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T13:40:18.032882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044036699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.080508+0000","last_change":"2026-02-26T02:44:44.080508+0000","last_active":"2026-02-26T02:44:44.080508+0000","last_peered":"2026-02-26T02:44:44.080508+0000","last_clean":"2026-02-26T02:44:44.080508+0000","last_became_active":"2026-02-26T02:44:44.080233+0000","last_became_peered":"2026-02-26T02:44:44.080233+0000","last_unstale":"2026-02-26T02:44:44.080508+0000","last_undegraded":"2026-02-26T02:44:44.080508+0000","last_fullsized":"2026-02-26T02:44:44.080508+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T04:07:41.024651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035731199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"15'2","reported_seq":42,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.077881+0000","last_change":"2026-02-26T02:44:43.078398+0000","last_active":"2026-02-26T02:44:44.077881+0000","last_peered":"2026-02-26T02:44:44.077881+0000","last_clean":"2026-02-26T02:44:44.077881+0000","last_became_active":"2026-02-26T02:44:43.077765+0000","last_became_peered":"2026-02-26T02:44:43.077765+0000","last_unstale":"2026-02-26T02:44:44.077881+0000","last_undegraded":"2026-02-26T02:44:44.077881+0000","last_fullsized":"2026-02-26T02:44:44.077881+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T05:33:48.482483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027048099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.079977+0000","last_change":"2026-02-26T02:44:44.079977+0000","last_active":"2026-02-26T02:44:44.079977+0000","last_peered":"2026-02-26T02:44:44.079977+0000","last_clean":"2026-02-26T02:44:44.079977+0000","last_became_active":"2026-02-26T02:44:44.079664+0000","last_became_peered":"2026-02-26T02:44:44.079664+0000","last_unstale":"2026-02-26T02:44:44.079977+0000","last_undegraded":"2026-02-26T02:44:44.079977+0000","last_fullsized":"2026-02-26T02:44:44.079977+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T08:02:36.823659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.078837+0000","last_change":"2026-02-26T02:44:43.078034+0000","last_active":"2026-02-26T02:44:44.078837+0000","last_peered":"2026-02-26T02:44:44.078837+0000","last_clean":"2026-02-26T02:44:44.078837+0000","last_became_active":"2026-02-26T02:44:43.077702+0000","last_became_peered":"2026-02-26T02:44:43.077702+0000","last_unstale":"2026-02-26T02:44:44.078837+0000","last_undegraded":"2026-02-26T02:44:44.078837+0000","last_fullsized":"2026-02-26T02:44:44.078837+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T10:01:12.136568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036294300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31840,"kb_used_data":204,"kb_used_omap":4,"kb_used_meta":31611,"kb_avail":94340000,"statfs":{"total":96636764160,"available":96604160000,"internally_reserved":0,"allocated":208896,"data_stored":56499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4430,"internal_metadata":32370354},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247811,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":200,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":204800,"data_stored":56480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4431,"internal_metadata":28110513},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313221,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":204,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":208896,"data_stored":56499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4434,"internal_metadata":28110510},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T02:44:52.969 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-26T02:44:52.969 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-26T02:44:52.969 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-26T02:44:52.969 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-26T02:44:53.143 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:44:53.143 INFO:teuthology.orchestra.run.trial043.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-26T02:44:53.150 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-26T02:44:53.150 INFO:teuthology.run_tasks:Running task exec... 2026-02-26T02:44:53.159 INFO:teuthology.task.exec:Executing custom commands... 2026-02-26T02:44:53.159 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@trial043.front.sepia.ceph.com 2026-02-26T02:44:53.159 DEBUG:teuthology.orchestra.run.trial043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'find $TESTDIR/archive/coredump -type f -exec rm -f {} \;' 2026-02-26T02:44:53.183 DEBUG:teuthology.run_tasks:Unwinding manager ceph.restart 2026-02-26T02:44:53.194 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-26T02:44:53.210 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-26T02:44:53.210 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T02:44:53.377 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:44:53.378 INFO:teuthology.orchestra.run.trial043.stderr:dumped all 2026-02-26T02:44:53.385 INFO:teuthology.orchestra.run.trial043.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2026-02-26T02:44:51.654362+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":87208,"kb_used_data":608,"kb_used_omap":12,"kb_used_meta":86515,"kb_avail":283028312,"statfs":{"total":289910292480,"available":289820991488,"internally_reserved":0,"allocated":622592,"data_stored":169478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13295,"internal_metadata":88591377},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.579687"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":38,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.080071+0000","last_change":"2026-02-26T02:44:44.080071+0000","last_active":"2026-02-26T02:44:44.080071+0000","last_peered":"2026-02-26T02:44:44.080071+0000","last_clean":"2026-02-26T02:44:44.080071+0000","last_became_active":"2026-02-26T02:44:44.079848+0000","last_became_peered":"2026-02-26T02:44:44.079848+0000","last_unstale":"2026-02-26T02:44:44.080071+0000","last_undegraded":"2026-02-26T02:44:44.080071+0000","last_fullsized":"2026-02-26T02:44:44.080071+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T06:53:14.722053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047321700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.078948+0000","last_change":"2026-02-26T02:44:43.077939+0000","last_active":"2026-02-26T02:44:44.078948+0000","last_peered":"2026-02-26T02:44:44.078948+0000","last_clean":"2026-02-26T02:44:44.078948+0000","last_became_active":"2026-02-26T02:44:43.077596+0000","last_became_peered":"2026-02-26T02:44:43.077596+0000","last_unstale":"2026-02-26T02:44:44.078948+0000","last_undegraded":"2026-02-26T02:44:44.078948+0000","last_fullsized":"2026-02-26T02:44:44.078948+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T08:45:22.327951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055368699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.080409+0000","last_change":"2026-02-26T02:44:44.080409+0000","last_active":"2026-02-26T02:44:44.080409+0000","last_peered":"2026-02-26T02:44:44.080409+0000","last_clean":"2026-02-26T02:44:44.080409+0000","last_became_active":"2026-02-26T02:44:44.080002+0000","last_became_peered":"2026-02-26T02:44:44.080002+0000","last_unstale":"2026-02-26T02:44:44.080409+0000","last_undegraded":"2026-02-26T02:44:44.080409+0000","last_fullsized":"2026-02-26T02:44:44.080409+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T12:15:28.872976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.078909+0000","last_change":"2026-02-26T02:44:43.078220+0000","last_active":"2026-02-26T02:44:44.078909+0000","last_peered":"2026-02-26T02:44:44.078909+0000","last_clean":"2026-02-26T02:44:44.078909+0000","last_became_active":"2026-02-26T02:44:43.077809+0000","last_became_peered":"2026-02-26T02:44:43.077809+0000","last_unstale":"2026-02-26T02:44:44.078909+0000","last_undegraded":"2026-02-26T02:44:44.078909+0000","last_fullsized":"2026-02-26T02:44:44.078909+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T13:40:18.032882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044036699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.080508+0000","last_change":"2026-02-26T02:44:44.080508+0000","last_active":"2026-02-26T02:44:44.080508+0000","last_peered":"2026-02-26T02:44:44.080508+0000","last_clean":"2026-02-26T02:44:44.080508+0000","last_became_active":"2026-02-26T02:44:44.080233+0000","last_became_peered":"2026-02-26T02:44:44.080233+0000","last_unstale":"2026-02-26T02:44:44.080508+0000","last_undegraded":"2026-02-26T02:44:44.080508+0000","last_fullsized":"2026-02-26T02:44:44.080508+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T04:07:41.024651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035731199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"15'2","reported_seq":42,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.077881+0000","last_change":"2026-02-26T02:44:43.078398+0000","last_active":"2026-02-26T02:44:44.077881+0000","last_peered":"2026-02-26T02:44:44.077881+0000","last_clean":"2026-02-26T02:44:44.077881+0000","last_became_active":"2026-02-26T02:44:43.077765+0000","last_became_peered":"2026-02-26T02:44:43.077765+0000","last_unstale":"2026-02-26T02:44:44.077881+0000","last_undegraded":"2026-02-26T02:44:44.077881+0000","last_fullsized":"2026-02-26T02:44:44.077881+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T05:33:48.482483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027048099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.079977+0000","last_change":"2026-02-26T02:44:44.079977+0000","last_active":"2026-02-26T02:44:44.079977+0000","last_peered":"2026-02-26T02:44:44.079977+0000","last_clean":"2026-02-26T02:44:44.079977+0000","last_became_active":"2026-02-26T02:44:44.079664+0000","last_became_peered":"2026-02-26T02:44:44.079664+0000","last_unstale":"2026-02-26T02:44:44.079977+0000","last_undegraded":"2026-02-26T02:44:44.079977+0000","last_fullsized":"2026-02-26T02:44:44.079977+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T08:02:36.823659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.078837+0000","last_change":"2026-02-26T02:44:43.078034+0000","last_active":"2026-02-26T02:44:44.078837+0000","last_peered":"2026-02-26T02:44:44.078837+0000","last_clean":"2026-02-26T02:44:44.078837+0000","last_became_active":"2026-02-26T02:44:43.077702+0000","last_became_peered":"2026-02-26T02:44:43.077702+0000","last_unstale":"2026-02-26T02:44:44.078837+0000","last_undegraded":"2026-02-26T02:44:44.078837+0000","last_fullsized":"2026-02-26T02:44:44.078837+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T10:01:12.136568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036294300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31840,"kb_used_data":204,"kb_used_omap":4,"kb_used_meta":31611,"kb_avail":94340000,"statfs":{"total":96636764160,"available":96604160000,"internally_reserved":0,"allocated":208896,"data_stored":56499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4430,"internal_metadata":32370354},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247811,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":200,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":204800,"data_stored":56480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4431,"internal_metadata":28110513},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313221,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":204,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":208896,"data_stored":56499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4434,"internal_metadata":28110510},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T02:44:53.386 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T02:44:53.538 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:44:53.539 INFO:teuthology.orchestra.run.trial043.stderr:dumped all 2026-02-26T02:44:53.546 INFO:teuthology.orchestra.run.trial043.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2026-02-26T02:44:51.654362+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":87208,"kb_used_data":608,"kb_used_omap":12,"kb_used_meta":86515,"kb_avail":283028312,"statfs":{"total":289910292480,"available":289820991488,"internally_reserved":0,"allocated":622592,"data_stored":169478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13295,"internal_metadata":88591377},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.579687"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":38,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.080071+0000","last_change":"2026-02-26T02:44:44.080071+0000","last_active":"2026-02-26T02:44:44.080071+0000","last_peered":"2026-02-26T02:44:44.080071+0000","last_clean":"2026-02-26T02:44:44.080071+0000","last_became_active":"2026-02-26T02:44:44.079848+0000","last_became_peered":"2026-02-26T02:44:44.079848+0000","last_unstale":"2026-02-26T02:44:44.080071+0000","last_undegraded":"2026-02-26T02:44:44.080071+0000","last_fullsized":"2026-02-26T02:44:44.080071+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T06:53:14.722053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047321700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.078948+0000","last_change":"2026-02-26T02:44:43.077939+0000","last_active":"2026-02-26T02:44:44.078948+0000","last_peered":"2026-02-26T02:44:44.078948+0000","last_clean":"2026-02-26T02:44:44.078948+0000","last_became_active":"2026-02-26T02:44:43.077596+0000","last_became_peered":"2026-02-26T02:44:43.077596+0000","last_unstale":"2026-02-26T02:44:44.078948+0000","last_undegraded":"2026-02-26T02:44:44.078948+0000","last_fullsized":"2026-02-26T02:44:44.078948+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T08:45:22.327951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055368699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.080409+0000","last_change":"2026-02-26T02:44:44.080409+0000","last_active":"2026-02-26T02:44:44.080409+0000","last_peered":"2026-02-26T02:44:44.080409+0000","last_clean":"2026-02-26T02:44:44.080409+0000","last_became_active":"2026-02-26T02:44:44.080002+0000","last_became_peered":"2026-02-26T02:44:44.080002+0000","last_unstale":"2026-02-26T02:44:44.080409+0000","last_undegraded":"2026-02-26T02:44:44.080409+0000","last_fullsized":"2026-02-26T02:44:44.080409+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T12:15:28.872976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.078909+0000","last_change":"2026-02-26T02:44:43.078220+0000","last_active":"2026-02-26T02:44:44.078909+0000","last_peered":"2026-02-26T02:44:44.078909+0000","last_clean":"2026-02-26T02:44:44.078909+0000","last_became_active":"2026-02-26T02:44:43.077809+0000","last_became_peered":"2026-02-26T02:44:43.077809+0000","last_unstale":"2026-02-26T02:44:44.078909+0000","last_undegraded":"2026-02-26T02:44:44.078909+0000","last_fullsized":"2026-02-26T02:44:44.078909+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T13:40:18.032882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044036699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.080508+0000","last_change":"2026-02-26T02:44:44.080508+0000","last_active":"2026-02-26T02:44:44.080508+0000","last_peered":"2026-02-26T02:44:44.080508+0000","last_clean":"2026-02-26T02:44:44.080508+0000","last_became_active":"2026-02-26T02:44:44.080233+0000","last_became_peered":"2026-02-26T02:44:44.080233+0000","last_unstale":"2026-02-26T02:44:44.080508+0000","last_undegraded":"2026-02-26T02:44:44.080508+0000","last_fullsized":"2026-02-26T02:44:44.080508+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T04:07:41.024651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035731199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"15'2","reported_seq":42,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.077881+0000","last_change":"2026-02-26T02:44:43.078398+0000","last_active":"2026-02-26T02:44:44.077881+0000","last_peered":"2026-02-26T02:44:44.077881+0000","last_clean":"2026-02-26T02:44:44.077881+0000","last_became_active":"2026-02-26T02:44:43.077765+0000","last_became_peered":"2026-02-26T02:44:43.077765+0000","last_unstale":"2026-02-26T02:44:44.077881+0000","last_undegraded":"2026-02-26T02:44:44.077881+0000","last_fullsized":"2026-02-26T02:44:44.077881+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T05:33:48.482483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027048099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.079977+0000","last_change":"2026-02-26T02:44:44.079977+0000","last_active":"2026-02-26T02:44:44.079977+0000","last_peered":"2026-02-26T02:44:44.079977+0000","last_clean":"2026-02-26T02:44:44.079977+0000","last_became_active":"2026-02-26T02:44:44.079664+0000","last_became_peered":"2026-02-26T02:44:44.079664+0000","last_unstale":"2026-02-26T02:44:44.079977+0000","last_undegraded":"2026-02-26T02:44:44.079977+0000","last_fullsized":"2026-02-26T02:44:44.079977+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T08:02:36.823659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.078837+0000","last_change":"2026-02-26T02:44:43.078034+0000","last_active":"2026-02-26T02:44:44.078837+0000","last_peered":"2026-02-26T02:44:44.078837+0000","last_clean":"2026-02-26T02:44:44.078837+0000","last_became_active":"2026-02-26T02:44:43.077702+0000","last_became_peered":"2026-02-26T02:44:43.077702+0000","last_unstale":"2026-02-26T02:44:44.078837+0000","last_undegraded":"2026-02-26T02:44:44.078837+0000","last_fullsized":"2026-02-26T02:44:44.078837+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T10:01:12.136568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036294300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31840,"kb_used_data":204,"kb_used_omap":4,"kb_used_meta":31611,"kb_avail":94340000,"statfs":{"total":96636764160,"available":96604160000,"internally_reserved":0,"allocated":208896,"data_stored":56499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4430,"internal_metadata":32370354},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247811,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":200,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":204800,"data_stored":56480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4431,"internal_metadata":28110513},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313221,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":204,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":208896,"data_stored":56499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4434,"internal_metadata":28110510},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T02:44:53.547 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-26T02:44:53.548 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T02:44:54.620 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:44:54.620 INFO:teuthology.orchestra.run.trial043.stderr:dumped all 2026-02-26T02:44:54.631 INFO:teuthology.orchestra.run.trial043.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2026-02-26T02:44:53.654623+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":87208,"kb_used_data":608,"kb_used_omap":12,"kb_used_meta":86515,"kb_avail":283028312,"statfs":{"total":289910292480,"available":289820991488,"internally_reserved":0,"allocated":622592,"data_stored":169478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13295,"internal_metadata":88591377},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001103"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":38,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.080071+0000","last_change":"2026-02-26T02:44:44.080071+0000","last_active":"2026-02-26T02:44:44.080071+0000","last_peered":"2026-02-26T02:44:44.080071+0000","last_clean":"2026-02-26T02:44:44.080071+0000","last_became_active":"2026-02-26T02:44:44.079848+0000","last_became_peered":"2026-02-26T02:44:44.079848+0000","last_unstale":"2026-02-26T02:44:44.080071+0000","last_undegraded":"2026-02-26T02:44:44.080071+0000","last_fullsized":"2026-02-26T02:44:44.080071+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T06:53:14.722053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047321700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.078948+0000","last_change":"2026-02-26T02:44:43.077939+0000","last_active":"2026-02-26T02:44:44.078948+0000","last_peered":"2026-02-26T02:44:44.078948+0000","last_clean":"2026-02-26T02:44:44.078948+0000","last_became_active":"2026-02-26T02:44:43.077596+0000","last_became_peered":"2026-02-26T02:44:43.077596+0000","last_unstale":"2026-02-26T02:44:44.078948+0000","last_undegraded":"2026-02-26T02:44:44.078948+0000","last_fullsized":"2026-02-26T02:44:44.078948+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T08:45:22.327951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055368699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.080409+0000","last_change":"2026-02-26T02:44:44.080409+0000","last_active":"2026-02-26T02:44:44.080409+0000","last_peered":"2026-02-26T02:44:44.080409+0000","last_clean":"2026-02-26T02:44:44.080409+0000","last_became_active":"2026-02-26T02:44:44.080002+0000","last_became_peered":"2026-02-26T02:44:44.080002+0000","last_unstale":"2026-02-26T02:44:44.080409+0000","last_undegraded":"2026-02-26T02:44:44.080409+0000","last_fullsized":"2026-02-26T02:44:44.080409+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T12:15:28.872976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.078909+0000","last_change":"2026-02-26T02:44:43.078220+0000","last_active":"2026-02-26T02:44:44.078909+0000","last_peered":"2026-02-26T02:44:44.078909+0000","last_clean":"2026-02-26T02:44:44.078909+0000","last_became_active":"2026-02-26T02:44:43.077809+0000","last_became_peered":"2026-02-26T02:44:43.077809+0000","last_unstale":"2026-02-26T02:44:44.078909+0000","last_undegraded":"2026-02-26T02:44:44.078909+0000","last_fullsized":"2026-02-26T02:44:44.078909+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T13:40:18.032882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044036699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.080508+0000","last_change":"2026-02-26T02:44:44.080508+0000","last_active":"2026-02-26T02:44:44.080508+0000","last_peered":"2026-02-26T02:44:44.080508+0000","last_clean":"2026-02-26T02:44:44.080508+0000","last_became_active":"2026-02-26T02:44:44.080233+0000","last_became_peered":"2026-02-26T02:44:44.080233+0000","last_unstale":"2026-02-26T02:44:44.080508+0000","last_undegraded":"2026-02-26T02:44:44.080508+0000","last_fullsized":"2026-02-26T02:44:44.080508+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T04:07:41.024651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035731199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"15'2","reported_seq":42,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.077881+0000","last_change":"2026-02-26T02:44:43.078398+0000","last_active":"2026-02-26T02:44:44.077881+0000","last_peered":"2026-02-26T02:44:44.077881+0000","last_clean":"2026-02-26T02:44:44.077881+0000","last_became_active":"2026-02-26T02:44:43.077765+0000","last_became_peered":"2026-02-26T02:44:43.077765+0000","last_unstale":"2026-02-26T02:44:44.077881+0000","last_undegraded":"2026-02-26T02:44:44.077881+0000","last_fullsized":"2026-02-26T02:44:44.077881+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T05:33:48.482483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027048099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.079977+0000","last_change":"2026-02-26T02:44:44.079977+0000","last_active":"2026-02-26T02:44:44.079977+0000","last_peered":"2026-02-26T02:44:44.079977+0000","last_clean":"2026-02-26T02:44:44.079977+0000","last_became_active":"2026-02-26T02:44:44.079664+0000","last_became_peered":"2026-02-26T02:44:44.079664+0000","last_unstale":"2026-02-26T02:44:44.079977+0000","last_undegraded":"2026-02-26T02:44:44.079977+0000","last_fullsized":"2026-02-26T02:44:44.079977+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T08:02:36.823659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.078837+0000","last_change":"2026-02-26T02:44:43.078034+0000","last_active":"2026-02-26T02:44:44.078837+0000","last_peered":"2026-02-26T02:44:44.078837+0000","last_clean":"2026-02-26T02:44:44.078837+0000","last_became_active":"2026-02-26T02:44:43.077702+0000","last_became_peered":"2026-02-26T02:44:43.077702+0000","last_unstale":"2026-02-26T02:44:44.078837+0000","last_undegraded":"2026-02-26T02:44:44.078837+0000","last_fullsized":"2026-02-26T02:44:44.078837+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T10:01:12.136568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036294300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31840,"kb_used_data":204,"kb_used_omap":4,"kb_used_meta":31611,"kb_avail":94340000,"statfs":{"total":96636764160,"available":96604160000,"internally_reserved":0,"allocated":208896,"data_stored":56499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4430,"internal_metadata":32370354},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247812,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":200,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":204800,"data_stored":56480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4431,"internal_metadata":28110513},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313221,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":204,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":208896,"data_stored":56499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4434,"internal_metadata":28110510},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T02:44:54.631 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T02:44:54.788 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:44:54.788 INFO:teuthology.orchestra.run.trial043.stdout:{"epoch":24,"fsid":"6dea32ae-afd8-4e41-a06e-2ec4e4797235","created":"2026-02-26T02:42:40.023737+0000","modified":"2026-02-26T02:44:44.075883+0000","last_up_change":"2026-02-26T02:44:43.073961+0000","last_in_change":"2026-02-26T02:42:40.696050+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-26T02:42:54.301050+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3383d670-1454-4abb-8847-9dab847e0a5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":16,"up_from":21,"up_thru":22,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6801","nonce":2557732258}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6802","nonce":2557732258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6804","nonce":2557732258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6803","nonce":2557732258}]},"public_addr":"10.20.193.43:6801/2557732258","cluster_addr":"10.20.193.43:6802/2557732258","heartbeat_back_addr":"10.20.193.43:6804/2557732258","heartbeat_front_addr":"10.20.193.43:6803/2557732258","state":["exists","up"]},{"osd":1,"uuid":"729a8cd6-7014-40b4-9e02-940e867460f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":22,"up_thru":23,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6805","nonce":2010462298}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6806","nonce":2010462298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6808","nonce":2010462298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6807","nonce":2010462298}]},"public_addr":"10.20.193.43:6805/2010462298","cluster_addr":"10.20.193.43:6806/2010462298","heartbeat_back_addr":"10.20.193.43:6808/2010462298","heartbeat_front_addr":"10.20.193.43:6807/2010462298","state":["exists","up"]},{"osd":2,"uuid":"b1591962-eecc-4a4f-83d1-040bee63ba29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":16,"up_from":23,"up_thru":23,"down_at":19,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6809","nonce":2224833790}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6810","nonce":2224833790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6812","nonce":2224833790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"10.20.193.43:6811","nonce":2224833790}]},"public_addr":"10.20.193.43:6809/2224833790","cluster_addr":"10.20.193.43:6810/2224833790","heartbeat_back_addr":"10.20.193.43:6812/2224833790","heartbeat_front_addr":"10.20.193.43:6811/2224833790","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2026-02-26T02:44:33.052845+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T02:42:48.220681+0000","dead_epoch":0},{"osd":1,"down_stamp":"2026-02-26T02:44:34.056568+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T02:42:48.293491+0000","dead_epoch":0},{"osd":2,"down_stamp":"2026-02-26T02:44:35.059184+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-26T02:42:48.146728+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T02:44:55.797 INFO:tasks.ceph:Scrubbing osd.0 2026-02-26T02:44:55.797 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2026-02-26T02:44:55.876 INFO:teuthology.orchestra.run.trial043.stdout:{ 2026-02-26T02:44:55.876 INFO:teuthology.orchestra.run.trial043.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-26T02:44:55.876 INFO:teuthology.orchestra.run.trial043.stdout:} 2026-02-26T02:44:55.881 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2026-02-26T02:44:56.038 INFO:teuthology.orchestra.run.trial043.stderr:instructed osd(s) 0 to deep-scrub 2026-02-26T02:44:56.047 INFO:tasks.ceph:Scrubbing osd.1 2026-02-26T02:44:56.047 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2026-02-26T02:44:56.126 INFO:teuthology.orchestra.run.trial043.stdout:{ 2026-02-26T02:44:56.126 INFO:teuthology.orchestra.run.trial043.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-26T02:44:56.126 INFO:teuthology.orchestra.run.trial043.stdout:} 2026-02-26T02:44:56.132 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2026-02-26T02:44:56.288 INFO:teuthology.orchestra.run.trial043.stderr:instructed osd(s) 1 to deep-scrub 2026-02-26T02:44:56.295 INFO:tasks.ceph:Scrubbing osd.2 2026-02-26T02:44:56.295 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2026-02-26T02:44:56.372 INFO:teuthology.orchestra.run.trial043.stdout:{ 2026-02-26T02:44:56.373 INFO:teuthology.orchestra.run.trial043.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-26T02:44:56.373 INFO:teuthology.orchestra.run.trial043.stdout:} 2026-02-26T02:44:56.379 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2026-02-26T02:44:59.534 INFO:teuthology.orchestra.run.trial043.stderr:instructed osd(s) 2 to deep-scrub 2026-02-26T02:44:59.543 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T02:44:59.696 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:44:59.696 INFO:teuthology.orchestra.run.trial043.stderr:dumped all 2026-02-26T02:44:59.704 INFO:teuthology.orchestra.run.trial043.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2026-02-26T02:44:59.655489+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":87208,"kb_used_data":608,"kb_used_omap":12,"kb_used_meta":86515,"kb_avail":283028312,"statfs":{"total":289910292480,"available":289820991488,"internally_reserved":0,"allocated":622592,"data_stored":169478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13295,"internal_metadata":88591377},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001451"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":38,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.080071+0000","last_change":"2026-02-26T02:44:44.080071+0000","last_active":"2026-02-26T02:44:44.080071+0000","last_peered":"2026-02-26T02:44:44.080071+0000","last_clean":"2026-02-26T02:44:44.080071+0000","last_became_active":"2026-02-26T02:44:44.079848+0000","last_became_peered":"2026-02-26T02:44:44.079848+0000","last_unstale":"2026-02-26T02:44:44.080071+0000","last_undegraded":"2026-02-26T02:44:44.080071+0000","last_fullsized":"2026-02-26T02:44:44.080071+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T06:53:14.722053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047321700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.078948+0000","last_change":"2026-02-26T02:44:43.077939+0000","last_active":"2026-02-26T02:44:44.078948+0000","last_peered":"2026-02-26T02:44:44.078948+0000","last_clean":"2026-02-26T02:44:44.078948+0000","last_became_active":"2026-02-26T02:44:43.077596+0000","last_became_peered":"2026-02-26T02:44:43.077596+0000","last_unstale":"2026-02-26T02:44:44.078948+0000","last_undegraded":"2026-02-26T02:44:44.078948+0000","last_fullsized":"2026-02-26T02:44:44.078948+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T08:45:22.327951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055368699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.080409+0000","last_change":"2026-02-26T02:44:44.080409+0000","last_active":"2026-02-26T02:44:44.080409+0000","last_peered":"2026-02-26T02:44:44.080409+0000","last_clean":"2026-02-26T02:44:44.080409+0000","last_became_active":"2026-02-26T02:44:44.080002+0000","last_became_peered":"2026-02-26T02:44:44.080002+0000","last_unstale":"2026-02-26T02:44:44.080409+0000","last_undegraded":"2026-02-26T02:44:44.080409+0000","last_fullsized":"2026-02-26T02:44:44.080409+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T12:15:28.872976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.078909+0000","last_change":"2026-02-26T02:44:43.078220+0000","last_active":"2026-02-26T02:44:44.078909+0000","last_peered":"2026-02-26T02:44:44.078909+0000","last_clean":"2026-02-26T02:44:44.078909+0000","last_became_active":"2026-02-26T02:44:43.077809+0000","last_became_peered":"2026-02-26T02:44:43.077809+0000","last_unstale":"2026-02-26T02:44:44.078909+0000","last_undegraded":"2026-02-26T02:44:44.078909+0000","last_fullsized":"2026-02-26T02:44:44.078909+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T13:40:18.032882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044036699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.080508+0000","last_change":"2026-02-26T02:44:44.080508+0000","last_active":"2026-02-26T02:44:44.080508+0000","last_peered":"2026-02-26T02:44:44.080508+0000","last_clean":"2026-02-26T02:44:44.080508+0000","last_became_active":"2026-02-26T02:44:44.080233+0000","last_became_peered":"2026-02-26T02:44:44.080233+0000","last_unstale":"2026-02-26T02:44:44.080508+0000","last_undegraded":"2026-02-26T02:44:44.080508+0000","last_fullsized":"2026-02-26T02:44:44.080508+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T04:07:41.024651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035731199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"15'2","reported_seq":43,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.077948+0000","last_change":"2026-02-26T02:44:43.078398+0000","last_active":"2026-02-26T02:44:44.077948+0000","last_peered":"2026-02-26T02:44:44.077948+0000","last_clean":"2026-02-26T02:44:44.077948+0000","last_became_active":"2026-02-26T02:44:43.077765+0000","last_became_peered":"2026-02-26T02:44:43.077765+0000","last_unstale":"2026-02-26T02:44:44.077948+0000","last_undegraded":"2026-02-26T02:44:44.077948+0000","last_fullsized":"2026-02-26T02:44:44.077948+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027048099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.079977+0000","last_change":"2026-02-26T02:44:44.079977+0000","last_active":"2026-02-26T02:44:44.079977+0000","last_peered":"2026-02-26T02:44:44.079977+0000","last_clean":"2026-02-26T02:44:44.079977+0000","last_became_active":"2026-02-26T02:44:44.079664+0000","last_became_peered":"2026-02-26T02:44:44.079664+0000","last_unstale":"2026-02-26T02:44:44.079977+0000","last_undegraded":"2026-02-26T02:44:44.079977+0000","last_fullsized":"2026-02-26T02:44:44.079977+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T08:02:36.823659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:44.078837+0000","last_change":"2026-02-26T02:44:43.078034+0000","last_active":"2026-02-26T02:44:44.078837+0000","last_peered":"2026-02-26T02:44:44.078837+0000","last_clean":"2026-02-26T02:44:44.078837+0000","last_became_active":"2026-02-26T02:44:43.077702+0000","last_became_peered":"2026-02-26T02:44:43.077702+0000","last_unstale":"2026-02-26T02:44:44.078837+0000","last_undegraded":"2026-02-26T02:44:44.078837+0000","last_fullsized":"2026-02-26T02:44:44.078837+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:42:54.321984+0000","last_clean_scrub_stamp":"2026-02-26T02:42:54.321984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T10:01:12.136568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036294300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":22,"seq":94489280517,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31840,"kb_used_data":204,"kb_used_omap":4,"kb_used_meta":31611,"kb_avail":94340000,"statfs":{"total":96636764160,"available":96604160000,"internally_reserved":0,"allocated":208896,"data_stored":56499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4430,"internal_metadata":32370354},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247813,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":200,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":204800,"data_stored":56480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4431,"internal_metadata":28110513},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313222,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":204,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":208896,"data_stored":56499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4434,"internal_metadata":28110510},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T02:44:59.709 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2026-02-26T02:42:54.321984+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=2, tm_min=42, tm_sec=54, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=2, tm_min=44, tm_sec=54, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T02:44:59.709 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2026-02-26T02:42:54.321984+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=2, tm_min=42, tm_sec=54, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=2, tm_min=44, tm_sec=54, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T02:44:59.709 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2026-02-26T02:42:54.321984+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=2, tm_min=42, tm_sec=54, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=2, tm_min=44, tm_sec=54, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T02:44:59.709 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2026-02-26T02:42:54.321984+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=2, tm_min=42, tm_sec=54, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=2, tm_min=44, tm_sec=54, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T02:44:59.709 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2026-02-26T02:42:54.321984+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=2, tm_min=42, tm_sec=54, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=2, tm_min=44, tm_sec=54, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T02:44:59.710 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2026-02-26T02:42:54.321984+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=2, tm_min=42, tm_sec=54, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=2, tm_min=44, tm_sec=54, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T02:44:59.710 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2026-02-26T02:42:54.321984+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=2, tm_min=42, tm_sec=54, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=2, tm_min=44, tm_sec=54, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T02:44:59.710 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2026-02-26T02:42:54.321984+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=2, tm_min=42, tm_sec=54, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=2, tm_min=44, tm_sec=54, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T02:44:59.710 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2026-02-26T02:45:19.711 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T02:45:19.865 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-26T02:45:19.865 INFO:teuthology.orchestra.run.trial043.stderr:dumped all 2026-02-26T02:45:19.873 INFO:teuthology.orchestra.run.trial043.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2026-02-26T02:45:19.657134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":87184,"kb_used_data":608,"kb_used_omap":12,"kb_used_meta":86515,"kb_avail":283028336,"statfs":{"total":289910292480,"available":289821016064,"internally_reserved":0,"allocated":622592,"data_stored":169478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13295,"internal_metadata":88591377},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.000872"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":47,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:59.584761+0000","last_change":"2026-02-26T02:44:59.584772+0000","last_active":"2026-02-26T02:44:59.584761+0000","last_peered":"2026-02-26T02:44:59.584761+0000","last_clean":"2026-02-26T02:44:59.584761+0000","last_became_active":"2026-02-26T02:44:44.079848+0000","last_became_peered":"2026-02-26T02:44:44.079848+0000","last_unstale":"2026-02-26T02:44:59.584761+0000","last_undegraded":"2026-02-26T02:44:59.584761+0000","last_fullsized":"2026-02-26T02:44:59.584761+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:44:59.584725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:44:59.584725+0000","last_clean_scrub_stamp":"2026-02-26T02:44:59.584725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T14:12:24.237494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047321700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":48,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:56.609245+0000","last_change":"2026-02-26T02:44:56.609258+0000","last_active":"2026-02-26T02:44:56.609245+0000","last_peered":"2026-02-26T02:44:56.609245+0000","last_clean":"2026-02-26T02:44:56.609245+0000","last_became_active":"2026-02-26T02:44:43.077596+0000","last_became_peered":"2026-02-26T02:44:43.077596+0000","last_unstale":"2026-02-26T02:44:56.609245+0000","last_undegraded":"2026-02-26T02:44:56.609245+0000","last_fullsized":"2026-02-26T02:44:56.609245+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:44:56.609196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:44:56.609196+0000","last_clean_scrub_stamp":"2026-02-26T02:44:56.609196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T10:12:26.245570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055368699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":46,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:45:01.245332+0000","last_change":"2026-02-26T02:45:01.245342+0000","last_active":"2026-02-26T02:45:01.245332+0000","last_peered":"2026-02-26T02:45:01.245332+0000","last_clean":"2026-02-26T02:45:01.245332+0000","last_became_active":"2026-02-26T02:44:44.080002+0000","last_became_peered":"2026-02-26T02:44:44.080002+0000","last_unstale":"2026-02-26T02:45:01.245332+0000","last_undegraded":"2026-02-26T02:45:01.245332+0000","last_fullsized":"2026-02-26T02:45:01.245332+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:45:01.245294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:45:01.245294+0000","last_clean_scrub_stamp":"2026-02-26T02:45:01.245294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T08:56:32.634614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":48,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:58.617595+0000","last_change":"2026-02-26T02:44:58.617605+0000","last_active":"2026-02-26T02:44:58.617595+0000","last_peered":"2026-02-26T02:44:58.617595+0000","last_clean":"2026-02-26T02:44:58.617595+0000","last_became_active":"2026-02-26T02:44:43.077809+0000","last_became_peered":"2026-02-26T02:44:43.077809+0000","last_unstale":"2026-02-26T02:44:58.617595+0000","last_undegraded":"2026-02-26T02:44:58.617595+0000","last_fullsized":"2026-02-26T02:44:58.617595+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:44:58.617557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:44:58.617557+0000","last_clean_scrub_stamp":"2026-02-26T02:44:58.617557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T12:07:27.547412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044036699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"24'2","reported_seq":49,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:45:00.613352+0000","last_change":"2026-02-26T02:45:00.613366+0000","last_active":"2026-02-26T02:45:00.613352+0000","last_peered":"2026-02-26T02:45:00.613352+0000","last_clean":"2026-02-26T02:45:00.613352+0000","last_became_active":"2026-02-26T02:44:44.080233+0000","last_became_peered":"2026-02-26T02:44:44.080233+0000","last_unstale":"2026-02-26T02:45:00.613352+0000","last_undegraded":"2026-02-26T02:45:00.613352+0000","last_fullsized":"2026-02-26T02:45:00.613352+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'2","last_scrub_stamp":"2026-02-26T02:45:00.613316+0000","last_deep_scrub":"24'2","last_deep_scrub_stamp":"2026-02-26T02:45:00.613316+0000","last_clean_scrub_stamp":"2026-02-26T02:45:00.613316+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T05:09:11.538676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035731199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"24'3","reported_seq":52,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:56.584520+0000","last_change":"2026-02-26T02:44:56.584565+0000","last_active":"2026-02-26T02:44:56.584520+0000","last_peered":"2026-02-26T02:44:56.584520+0000","last_clean":"2026-02-26T02:44:56.584520+0000","last_became_active":"2026-02-26T02:44:43.077765+0000","last_became_peered":"2026-02-26T02:44:43.077765+0000","last_unstale":"2026-02-26T02:44:56.584520+0000","last_undegraded":"2026-02-26T02:44:56.584520+0000","last_fullsized":"2026-02-26T02:44:56.584520+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'3","last_scrub_stamp":"2026-02-26T02:44:56.584433+0000","last_deep_scrub":"24'3","last_deep_scrub_stamp":"2026-02-26T02:44:56.584433+0000","last_clean_scrub_stamp":"2026-02-26T02:44:56.584433+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T05:52:34.256448+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00027048099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":46,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:45:00.221812+0000","last_change":"2026-02-26T02:45:00.221837+0000","last_active":"2026-02-26T02:45:00.221812+0000","last_peered":"2026-02-26T02:45:00.221812+0000","last_clean":"2026-02-26T02:45:00.221812+0000","last_became_active":"2026-02-26T02:44:44.079664+0000","last_became_peered":"2026-02-26T02:44:44.079664+0000","last_unstale":"2026-02-26T02:45:00.221812+0000","last_undegraded":"2026-02-26T02:45:00.221812+0000","last_fullsized":"2026-02-26T02:45:00.221812+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:45:00.221741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:45:00.221741+0000","last_clean_scrub_stamp":"2026-02-26T02:45:00.221741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T12:58:56.316143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":48,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-26T02:44:57.634358+0000","last_change":"2026-02-26T02:44:57.634382+0000","last_active":"2026-02-26T02:44:57.634358+0000","last_peered":"2026-02-26T02:44:57.634358+0000","last_clean":"2026-02-26T02:44:57.634358+0000","last_became_active":"2026-02-26T02:44:43.077702+0000","last_became_peered":"2026-02-26T02:44:43.077702+0000","last_unstale":"2026-02-26T02:44:57.634358+0000","last_undegraded":"2026-02-26T02:44:57.634358+0000","last_fullsized":"2026-02-26T02:44:57.634358+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T02:44:57.634284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T02:44:57.634284+0000","last_clean_scrub_stamp":"2026-02-26T02:44:57.634284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-27T03:47:24.668377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036294300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":22,"seq":94489280521,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31836,"kb_used_data":204,"kb_used_omap":4,"kb_used_meta":31611,"kb_avail":94340004,"statfs":{"total":96636764160,"available":96604164096,"internally_reserved":0,"allocated":208896,"data_stored":56499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4430,"internal_metadata":32370354},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247817,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":200,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":204800,"data_stored":56480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4431,"internal_metadata":28110513},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":21,"seq":90194313226,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27676,"kb_used_data":204,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":94344164,"statfs":{"total":96636764160,"available":96608423936,"internally_reserved":0,"allocated":208896,"data_stored":56499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4434,"internal_metadata":28110510},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T02:45:19.875 DEBUG:teuthology.orchestra.run.trial043:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-26T02:45:20.041 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-26T02:45:20.041 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-26T02:45:20.041 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-26T02:45:20.041 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T02:45:20.078 INFO:tasks.ceph.osd.0:Stopped 2026-02-26T02:45:20.078 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-26T02:45:20.078 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T02:45:20.162 INFO:tasks.ceph.osd.1:Stopped 2026-02-26T02:45:20.162 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-26T02:45:20.162 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T02:45:20.196 INFO:tasks.ceph.osd.2:Stopped 2026-02-26T02:45:20.196 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-26T02:45:20.196 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-26T02:45:20.196 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T02:45:20.267 INFO:tasks.ceph.mgr.x:Stopped 2026-02-26T02:45:20.267 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-26T02:45:20.267 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-26T02:45:20.267 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T02:45:20.317 INFO:tasks.ceph.mon.a:Stopped 2026-02-26T02:45:20.317 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-26T02:45:20.318 DEBUG:teuthology.orchestra.run.trial043:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'Reduced data availability' | egrep -v 'OSD_.*DOWN' | egrep -v '\(RECENT_CRASH\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v '\(MON_DOWN\)' | head -n 1 2026-02-26T02:45:20.382 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial043.front.sepia.ceph.com 2026-02-26T02:45:20.382 DEBUG:teuthology.orchestra.run.trial043:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-26T02:45:20.452 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial043.front.sepia.ceph.com 2026-02-26T02:45:20.452 DEBUG:teuthology.orchestra.run.trial043:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-26T02:45:20.519 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial043.front.sepia.ceph.com 2026-02-26T02:45:20.519 DEBUG:teuthology.orchestra.run.trial043:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-26T02:45:20.585 INFO:tasks.ceph:Archiving mon data... 2026-02-26T02:45:20.588 DEBUG:teuthology.misc:Transferring archived files from trial043:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/yuriw-2026-02-26_01:51:16-rados-wip-yuri9-testing-2026-02-25-1600-squid-distro-default-trial/71042/data/mon.a.tgz 2026-02-26T02:45:20.588 DEBUG:teuthology.orchestra.run.trial043:> mktemp 2026-02-26T02:45:20.641 INFO:teuthology.orchestra.run.trial043.stdout:/tmp/tmp.rzCnAr2om8 2026-02-26T02:45:20.641 DEBUG:teuthology.orchestra.run.trial043:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.rzCnAr2om8 2026-02-26T02:45:20.725 DEBUG:teuthology.orchestra.run.trial043:> sudo chmod 0666 /tmp/tmp.rzCnAr2om8 2026-02-26T02:45:20.804 DEBUG:teuthology.orchestra.remote:trial043:/tmp/tmp.rzCnAr2om8 is 153KB 2026-02-26T02:45:20.823 DEBUG:teuthology.orchestra.run.trial043:> rm -fr /tmp/tmp.rzCnAr2om8 2026-02-26T02:45:20.839 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-26T02:45:20.840 DEBUG:teuthology.orchestra.run.trial043:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-26T02:45:20.959 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-26T02:45:20.959 INFO:tasks.ceph:Archiving crash dumps... 2026-02-26T02:45:20.961 DEBUG:teuthology.misc:Transferring archived files from trial043:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/yuriw-2026-02-26_01:51:16-rados-wip-yuri9-testing-2026-02-25-1600-squid-distro-default-trial/71042/remote/trial043/crash 2026-02-26T02:45:20.963 DEBUG:teuthology.orchestra.run.trial043:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-26T02:45:21.103 INFO:tasks.ceph:Compressing logs... 2026-02-26T02:45:21.104 DEBUG:teuthology.orchestra.run.trial043:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-26T02:45:21.160 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-26T02:45:21.160 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43150.log 2026-02-26T02:45:21.160 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46419.log 2026-02-26T02:45:21.161 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.43150.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-26T02:45:21.161 INFO:teuthology.orchestra.run.trial043.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43150.log.gz 2026-02-26T02:45:21.161 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.crash.trial043.47489.log 2026-02-26T02:45:21.161 INFO:teuthology.orchestra.run.trial043.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46419.log.gz 2026-02-26T02:45:21.161 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51886.log 2026-02-26T02:45:21.161 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.crash.trial043.47489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46618.log 2026-02-26T02:45:21.161 INFO:teuthology.orchestra.run.trial043.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.crash.trial043.47489.log.gz 2026-02-26T02:45:21.162 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5/var/log/ceph/ceph-client.admin.51886.log: --verbose -- /var/log/ceph/ceph-client.admin.46517.log 2026-02-26T02:45:21.162 INFO:teuthology.orchestra.run.trial043.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51886.log.gz 2026-02-26T02:45:21.162 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46618.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46618.log.gz 2026-02-26T02:45:21.162 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51224.log 2026-02-26T02:45:21.163 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46517.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-26T02:45:21.163 INFO:teuthology.orchestra.run.trial043.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46517.log.gz 2026-02-26T02:45:21.163 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.51224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47688.log 2026-02-26T02:45:21.163 INFO:teuthology.orchestra.run.trial043.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51224.log.gz 2026-02-26T02:45:21.163 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.crash.trial043.47585.log 2026-02-26T02:45:21.163 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47688.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47688.log.gz 2026-02-26T02:45:21.163 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51739.log 2026-02-26T02:45:21.163 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.crash.trial043.47585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47939.log 2026-02-26T02:45:21.164 INFO:teuthology.orchestra.run.trial043.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.crash.trial043.47585.log.gz 2026-02-26T02:45:21.164 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51420.log 2026-02-26T02:45:21.164 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.51739.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51739.log.gz 2026-02-26T02:45:21.164 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47939.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.47939.log.gz -5 2026-02-26T02:45:21.164 INFO:teuthology.orchestra.run.trial043.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43514.log 2026-02-26T02:45:21.164 INFO:teuthology.orchestra.run.trial043.stderr:gzip/var/log/ceph/ceph-client.admin.51420.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.47864.log 2026-02-26T02:45:21.165 INFO:teuthology.orchestra.run.trial043.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51420.log.gz 2026-02-26T02:45:21.165 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.43514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50983.log 2026-02-26T02:45:21.165 INFO:teuthology.orchestra.run.trial043.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43514.log.gz 2026-02-26T02:45:21.165 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47864.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47864.log.gz 2026-02-26T02:45:21.165 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46329.log 2026-02-26T02:45:21.165 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.50983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47007.log 2026-02-26T02:45:21.165 INFO:teuthology.orchestra.run.trial043.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50983.log.gz 2026-02-26T02:45:21.166 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51641.log 2026-02-26T02:45:21.166 INFO:teuthology.orchestra.run.trial043.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46329.log.gz 2026-02-26T02:45:21.166 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47007.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47007.log.gz 2026-02-26T02:45:21.166 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.crash.47597.log 2026-02-26T02:45:21.166 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.51641.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51641.log.gz 2026-02-26T02:45:21.166 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48033.log 2026-02-26T02:45:21.166 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.crash.47597.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.crash.47597.log.gz -- 2026-02-26T02:45:21.166 INFO:teuthology.orchestra.run.trial043.stderr: /var/log/ceph/ceph-client.admin.43038.log 2026-02-26T02:45:21.166 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.48033.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48033.log.gz 2026-02-26T02:45:21.167 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50881.log 2026-02-26T02:45:21.167 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.43038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43293.log 2026-02-26T02:45:21.167 INFO:teuthology.orchestra.run.trial043.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43038.log.gz 2026-02-26T02:45:21.167 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.50881.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50881.log.gz 2026-02-26T02:45:21.167 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47154.log 2026-02-26T02:45:21.168 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.43293.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43293.log.gz 2026-02-26T02:45:21.168 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46620.log 2026-02-26T02:45:21.168 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47154.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47154.log.gz 2026-02-26T02:45:21.168 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.38386.log 2026-02-26T02:45:21.168 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46620.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46620.log.gz 2026-02-26T02:45:21.168 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51935.log 2026-02-26T02:45:21.169 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph.tmp-client.admin.38386.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.38386.log.gz 2026-02-26T02:45:21.169 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47609.log 2026-02-26T02:45:21.169 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.51935.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51935.log.gz 2026-02-26T02:45:21.169 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43340.log 2026-02-26T02:45:21.169 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47609.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47609.log.gz 2026-02-26T02:45:21.169 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48732.log 2026-02-26T02:45:21.169 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.43340.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.50982.log -- replaced with /var/log/ceph/ceph-client.admin.43340.log.gz 2026-02-26T02:45:21.169 INFO:teuthology.orchestra.run.trial043.stderr: 2026-02-26T02:45:21.169 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.48732.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48732.log.gz 2026-02-26T02:45:21.169 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46468.log 2026-02-26T02:45:21.170 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.50982.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50982.log.gz 2026-02-26T02:45:21.170 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51592.log 2026-02-26T02:45:21.170 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46468.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46468.log.gz 2026-02-26T02:45:21.170 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51984.log 2026-02-26T02:45:21.170 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.51592.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51592.log.gz 2026-02-26T02:45:21.170 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47105.log 2026-02-26T02:45:21.170 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.51984.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51984.log.gz 2026-02-26T02:45:21.170 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46767.log 2026-02-26T02:45:21.170 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47105.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47105.log.gz 2026-02-26T02:45:21.170 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50783.log 2026-02-26T02:45:21.170 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46767.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46767.log.gz 2026-02-26T02:45:21.170 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51135.log 2026-02-26T02:45:21.171 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.50783.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50783.log.gz 2026-02-26T02:45:21.171 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46923.log 2026-02-26T02:45:21.172 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.51135.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51135.log.gz 2026-02-26T02:45:21.172 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52033.log 2026-02-26T02:45:21.172 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46923.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46923.log.gz 2026-02-26T02:45:21.172 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51469.log 2026-02-26T02:45:21.172 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.52033.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52033.log.gz 2026-02-26T02:45:21.172 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.crash.trial043.47537.log 2026-02-26T02:45:21.172 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.51469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46860.log 2026-02-26T02:45:21.172 INFO:teuthology.orchestra.run.trial043.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51469.log.gz 2026-02-26T02:45:21.172 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.crash.trial043.47537.log: 0.0% -- replaced with /var/log/ceph/ceph-client.crash.trial043.47537.log.gz 2026-02-26T02:45:21.172 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46765.log 2026-02-26T02:45:21.173 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46860.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46860.log.gz 2026-02-26T02:45:21.173 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51282.log 2026-02-26T02:45:21.173 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46765.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46765.log.gz 2026-02-26T02:45:21.173 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47633.log 2026-02-26T02:45:21.173 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.51282.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51282.log.gz 2026-02-26T02:45:21.173 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51690.log 2026-02-26T02:45:21.173 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47633.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-26T02:45:21.173 INFO:teuthology.orchestra.run.trial043.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47633.log.gz 2026-02-26T02:45:21.173 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.51690.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51690.log.gz 2026-02-26T02:45:21.173 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46766.log 2026-02-26T02:45:21.173 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47465.log 2026-02-26T02:45:21.174 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46766.log: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-26T02:45:21.175 INFO:teuthology.orchestra.run.trial043.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46766.log.gz 2026-02-26T02:45:21.175 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46259.log 2026-02-26T02:45:21.175 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52130.log 2026-02-26T02:45:21.175 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47465.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47465.log.gz 2026-02-26T02:45:21.175 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46259.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46259.log.gz 2026-02-26T02:45:21.175 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-26T02:45:21.175 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.52130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47561.log 2026-02-26T02:45:21.175 INFO:teuthology.orchestra.run.trial043.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52130.log.gz 2026-02-26T02:45:21.175 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48206.log 2026-02-26T02:45:21.175 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47561.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47561.log.gz 2026-02-26T02:45:21.175 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51279.log 2026-02-26T02:45:21.176 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.48206.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48206.log.gz 2026-02-26T02:45:21.176 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47663.log 2026-02-26T02:45:21.176 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.51279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51837.log 2026-02-26T02:45:21.176 INFO:teuthology.orchestra.run.trial043.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51279.log.gz 2026-02-26T02:45:21.176 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47663.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47663.log.gz 2026-02-26T02:45:21.176 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51104.log 2026-02-26T02:45:21.176 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.51837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47513.log 2026-02-26T02:45:21.176 INFO:teuthology.orchestra.run.trial043.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51837.log.gz 2026-02-26T02:45:21.176 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.51104.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51104.log.gz 2026-02-26T02:45:21.176 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51103.log 2026-02-26T02:45:21.177 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50832.log 2026-02-26T02:45:21.177 INFO:teuthology.orchestra.run.trial043.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47513.log.gz 2026-02-26T02:45:21.177 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.51103.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51103.log.gz 2026-02-26T02:45:21.177 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43387.log 2026-02-26T02:45:21.177 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.50832.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50832.log.gz 2026-02-26T02:45:21.177 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43582.log 2026-02-26T02:45:21.177 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.43387.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-26T02:45:21.177 INFO:teuthology.orchestra.run.trial043.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43387.log.gz 2026-02-26T02:45:21.177 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.43582.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43582.log.gz 2026-02-26T02:45:21.177 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.crash.47501.log 2026-02-26T02:45:21.178 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51543.log 2026-02-26T02:45:21.178 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.crash.47501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51371.log 2026-02-26T02:45:21.178 INFO:teuthology.orchestra.run.trial043.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.crash.47501.log.gz 2026-02-26T02:45:21.178 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.51543.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51543.log.gz 2026-02-26T02:45:21.178 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46619.log 2026-02-26T02:45:21.179 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.51371.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-26T02:45:21.179 INFO:teuthology.orchestra.run.trial043.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51371.log.gz 2026-02-26T02:45:21.179 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46619.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.46619.log.gz --verbose 2026-02-26T02:45:21.179 INFO:teuthology.orchestra.run.trial043.stderr: -- /var/log/ceph/ceph-client.admin.47056.log 2026-02-26T02:45:21.179 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48138.log 2026-02-26T02:45:21.179 INFO:teuthology.orchestra.run.trial043.stderr: 85.0% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-26T02:45:21.179 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50984.log 2026-02-26T02:45:21.179 INFO:teuthology.orchestra.run.trial043.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47056.log.gz 2026-02-26T02:45:21.180 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.48138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48138.log.gz 2026-02-26T02:45:21.180 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47714.log 2026-02-26T02:45:21.180 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.50984.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50984.log.gz 2026-02-26T02:45:21.180 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52083.log 2026-02-26T02:45:21.180 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51788.log 2026-02-26T02:45:21.180 INFO:teuthology.orchestra.run.trial043.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47714.log.gz 2026-02-26T02:45:21.180 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.52083.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52083.log.gz 2026-02-26T02:45:21.180 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46927.log 2026-02-26T02:45:21.181 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.51788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.crash.47549.log 2026-02-26T02:45:21.181 INFO:teuthology.orchestra.run.trial043.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51788.log.gz 2026-02-26T02:45:21.181 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46927.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46927.log.gz 2026-02-26T02:45:21.181 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46211.log 2026-02-26T02:45:21.181 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.crash.47549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46354.log 2026-02-26T02:45:21.181 INFO:teuthology.orchestra.run.trial043.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.crash.47549.log.gz 2026-02-26T02:45:21.181 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46211.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46211.log.gz 2026-02-26T02:45:21.182 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46354.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46354.log.gz 2026-02-26T02:45:21.192 INFO:teuthology.orchestra.run.trial043.stderr: 90.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-26T02:45:21.199 INFO:teuthology.orchestra.run.trial043.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-26T02:45:21.208 INFO:teuthology.orchestra.run.trial043.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-26T02:45:21.492 INFO:teuthology.orchestra.run.trial043.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-26T02:45:21.499 INFO:teuthology.orchestra.run.trial043.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-26T02:45:21.501 INFO:teuthology.orchestra.run.trial043.stderr: 2026-02-26T02:45:21.501 INFO:teuthology.orchestra.run.trial043.stderr:real 0m0.349s 2026-02-26T02:45:21.501 INFO:teuthology.orchestra.run.trial043.stderr:user 0m0.749s 2026-02-26T02:45:21.501 INFO:teuthology.orchestra.run.trial043.stderr:sys 0m0.072s 2026-02-26T02:45:21.502 INFO:tasks.ceph:Archiving logs... 2026-02-26T02:45:21.502 DEBUG:teuthology.misc:Transferring archived files from trial043:/var/log/ceph to /home/teuthworker/mnt/teuthology/yuriw-2026-02-26_01:51:16-rados-wip-yuri9-testing-2026-02-25-1600-squid-distro-default-trial/71042/remote/trial043/log 2026-02-26T02:45:21.507 DEBUG:teuthology.orchestra.run.trial043:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-26T02:45:21.699 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-26T02:45:21.707 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-26T02:45:21.707 DEBUG:teuthology.orchestra.run.trial043:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-26T02:45:21.803 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-26T02:45:21.814 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-26T02:45:21.814 DEBUG:teuthology.orchestra.run.trial043:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-26T02:45:21.831 INFO:teuthology.orchestra.run.trial043.stderr:bash: line 1: ntpq: command not found 2026-02-26T02:45:21.861 INFO:teuthology.orchestra.run.trial043.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-26T02:45:21.861 INFO:teuthology.orchestra.run.trial043.stdout:=============================================================================== 2026-02-26T02:45:21.861 INFO:teuthology.orchestra.run.trial043.stdout:^+ LAX.CALTICK.NET 2 6 177 49 -889us[ -889us] +/- 33ms 2026-02-26T02:45:21.861 INFO:teuthology.orchestra.run.trial043.stdout:^+ ntp.maxhost.io 2 6 177 47 -40us[ -40us] +/- 55ms 2026-02-26T02:45:21.861 INFO:teuthology.orchestra.run.trial043.stdout:^+ ntp.maxhost.io 2 6 173 45 +90us[ +90us] +/- 55ms 2026-02-26T02:45:21.861 INFO:teuthology.orchestra.run.trial043.stdout:^* 155.248.196.28 2 6 177 49 -87us[ -21us] +/- 49ms 2026-02-26T02:45:21.862 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-26T02:45:21.873 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-26T02:45:21.873 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-26T02:45:21.887 DEBUG:teuthology.orchestra.run.trial043:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-26T02:45:21.956 DEBUG:teuthology.orchestra.run.trial043:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2026-02-26T02:45:21.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T02:45:21.982 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-26T02:45:21.994 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-26T02:45:22.009 INFO:teuthology.task.internal:Duration was 417.983983 seconds 2026-02-26T02:45:22.009 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-26T02:45:22.032 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-26T02:45:22.032 DEBUG:teuthology.orchestra.run.trial043:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-26T02:45:22.069 INFO:teuthology.orchestra.run.trial043.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-26T02:45:22.261 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-26T02:45:22.262 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial043.front.sepia.ceph.com 2026-02-26T02:45:22.262 DEBUG:teuthology.orchestra.run.trial043:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-26T02:45:22.319 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-26T02:45:22.319 DEBUG:teuthology.orchestra.run.trial043:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-26T02:45:22.419 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-26T02:45:22.419 DEBUG:teuthology.orchestra.run.trial043:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-26T02:45:22.441 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-26T02:45:22.441 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-26T02:45:22.441 INFO:teuthology.orchestra.run.trial043.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-26T02:45:22.442 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-26T02:45:22.442 INFO:teuthology.orchestra.run.trial043.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-26T02:45:22.452 INFO:teuthology.orchestra.run.trial043.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 81.7% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-26T02:45:22.454 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-26T02:45:22.472 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-26T02:45:22.472 DEBUG:teuthology.orchestra.run.trial043:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-26T02:45:22.518 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-26T02:45:22.530 DEBUG:teuthology.orchestra.run.trial043:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-26T02:45:22.582 INFO:teuthology.orchestra.run.trial043.stdout:kernel.core_pattern = core 2026-02-26T02:45:22.596 DEBUG:teuthology.orchestra.run.trial043:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-26T02:45:22.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T02:45:22.652 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-26T02:45:22.663 INFO:teuthology.task.internal:Transferring archived files... 2026-02-26T02:45:22.663 DEBUG:teuthology.misc:Transferring archived files from trial043:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/yuriw-2026-02-26_01:51:16-rados-wip-yuri9-testing-2026-02-25-1600-squid-distro-default-trial/71042/remote/trial043 2026-02-26T02:45:22.664 DEBUG:teuthology.orchestra.run.trial043:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-26T02:45:22.734 INFO:teuthology.task.internal:Removing archive directory... 2026-02-26T02:45:22.734 DEBUG:teuthology.orchestra.run.trial043:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-26T02:45:22.775 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-26T02:45:22.793 INFO:teuthology.task.internal:Not uploading archives. 2026-02-26T02:45:22.793 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-26T02:45:22.809 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-26T02:45:22.809 DEBUG:teuthology.orchestra.run.trial043:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-26T02:45:22.830 INFO:teuthology.orchestra.run.trial043.stdout: 920834 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 26 02:45 /home/ubuntu/cephtest 2026-02-26T02:45:22.831 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-26T02:45:22.842 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-26T02:45:22.879 INFO:teuthology.run:Summary data: description: rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} duration: 417.9839825630188 flavor: default owner: scheduled_yuriw@soko04.front.sepia.ceph.com success: true 2026-02-26T02:45:22.879 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-26T02:45:23.007 INFO:teuthology.run:pass