2024-09-10T19:33:12.664 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-10T19:33:12.684 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-10T19:33:12.762 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899503 branch: wip-bharath7-testing-2024-09-10-1409 description: rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{centos_latest}} email: skanta@redhat.com first_in_suite: false job_id: '7899503' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath7-testing-2024-09-10-1409 ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd debug verify cached snaps: true osd debug verify missing on start: true osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 workunit: branch: wip-bharath7-testing-2024-09-10-1409 sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 - osd.4 - osd.5 - osd.6 - osd.7 - osd.8 - osd.9 - client.0 seed: 5245 sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath7-testing-2024-09-10-1409 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 targets: smithi082.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHbr0sYVGCgNZ2RW/V356HQvfWiawJHDCfB9WjUEtOWpfqTyF341DGVdKqOF2ELmlzDkOW6ozg56Rf8gMUeBwh8= tasks: - install: null - ceph: conf: osd: osd max object name len: 400 osd max object namespace len: 64 log-ignorelist: - but it is still running - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_APP_NOT_ENABLED\) pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force - workunit: clients: all: - rados/test_health_warnings.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_15:30:27 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-10T19:33:12.763 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa; will attempt to use it 2024-09-10T19:33:12.763 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks 2024-09-10T19:33:12.764 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-10T19:33:12.765 INFO:teuthology.task.internal:Checking packages... 2024-09-10T19:33:12.787 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46' 2024-09-10T19:33:12.787 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-10T19:33:12.787 INFO:teuthology.packaging:ref: None 2024-09-10T19:33:12.788 INFO:teuthology.packaging:tag: None 2024-09-10T19:33:12.788 INFO:teuthology.packaging:branch: wip-bharath7-testing-2024-09-10-1409 2024-09-10T19:33:12.788 INFO:teuthology.packaging:sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T19:33:12.788 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath7-testing-2024-09-10-1409 2024-09-10T19:33:12.927 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4858.gbad1f558 2024-09-10T19:33:12.929 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-10T19:33:12.937 INFO:teuthology.task.internal:no buildpackages task found 2024-09-10T19:33:12.937 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-10T19:33:12.968 INFO:teuthology.task.internal:Saving configuration 2024-09-10T19:33:12.980 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-10T19:33:12.986 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-10T19:33:13.011 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi082.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899503', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-10 19:28:28.239189', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHbr0sYVGCgNZ2RW/V356HQvfWiawJHDCfB9WjUEtOWpfqTyF341DGVdKqOF2ELmlzDkOW6ozg56Rf8gMUeBwh8='} 2024-09-10T19:33:13.012 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-10T19:33:13.018 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'osd.8', 'osd.9', 'client.0'] 2024-09-10T19:33:13.018 INFO:teuthology.run_tasks:Running task console_log... 2024-09-10T19:33:13.065 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc68e513be0>, signals=[15]) 2024-09-10T19:33:13.065 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-10T19:33:13.071 INFO:teuthology.task.internal:Opening connections... 2024-09-10T19:33:13.071 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi082.front.sepia.ceph.com 2024-09-10T19:33:13.073 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T19:33:13.152 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-10T19:33:13.158 DEBUG:teuthology.orchestra.run.smithi082:> uname -m 2024-09-10T19:33:13.238 INFO:teuthology.orchestra.run.smithi082.stdout:x86_64 2024-09-10T19:33:13.239 DEBUG:teuthology.orchestra.run.smithi082:> cat /etc/os-release 2024-09-10T19:33:13.297 INFO:teuthology.orchestra.run.smithi082.stdout:NAME="CentOS Stream" 2024-09-10T19:33:13.297 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION="9" 2024-09-10T19:33:13.297 INFO:teuthology.orchestra.run.smithi082.stdout:ID="centos" 2024-09-10T19:33:13.297 INFO:teuthology.orchestra.run.smithi082.stdout:ID_LIKE="rhel fedora" 2024-09-10T19:33:13.297 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION_ID="9" 2024-09-10T19:33:13.297 INFO:teuthology.orchestra.run.smithi082.stdout:PLATFORM_ID="platform:el9" 2024-09-10T19:33:13.297 INFO:teuthology.orchestra.run.smithi082.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-10T19:33:13.297 INFO:teuthology.orchestra.run.smithi082.stdout:ANSI_COLOR="0;31" 2024-09-10T19:33:13.297 INFO:teuthology.orchestra.run.smithi082.stdout:LOGO="fedora-logo-icon" 2024-09-10T19:33:13.298 INFO:teuthology.orchestra.run.smithi082.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-10T19:33:13.298 INFO:teuthology.orchestra.run.smithi082.stdout:HOME_URL="https://centos.org/" 2024-09-10T19:33:13.298 INFO:teuthology.orchestra.run.smithi082.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-10T19:33:13.298 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-10T19:33:13.298 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-10T19:33:13.298 INFO:teuthology.lock.ops:Updating smithi082.front.sepia.ceph.com on lock server 2024-09-10T19:33:13.316 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-10T19:33:13.325 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-10T19:33:13.343 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-10T19:33:13.343 DEBUG:teuthology.orchestra.run.smithi082:> test '!' -e /home/ubuntu/cephtest 2024-09-10T19:33:13.360 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-10T19:33:13.367 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-10T19:33:13.367 DEBUG:teuthology.orchestra.run.smithi082:> test -z $(ls -A /var/lib/ceph) 2024-09-10T19:33:13.426 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-10T19:33:13.455 INFO:teuthology.run_tasks:Running task kernel... 2024-09-10T19:33:13.470 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-10T19:33:13.470 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-10T19:33:13.470 DEBUG:teuthology.orchestra.run.smithi082:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-10T19:33:13.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:33:13.487 DEBUG:teuthology.orchestra.run.smithi082:> uname -r 2024-09-10T19:33:13.545 INFO:teuthology.orchestra.run.smithi082.stdout:5.14.0-503.el9.x86_64 2024-09-10T19:33:13.545 INFO:teuthology.task.kernel:Running kernel on smithi082: 5.14.0-503.el9.x86_64 2024-09-10T19:33:13.545 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum install -y kernel 2024-09-10T19:33:15.806 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.2 MB 00:00 2024-09-10T19:33:17.791 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - AppStream 24 MB/s | 20 MB 00:00 2024-09-10T19:33:22.174 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-10T19:33:24.075 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 19 kB 00:00 2024-09-10T19:33:25.187 INFO:teuthology.orchestra.run.smithi082.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-10T19:33:30.625 INFO:teuthology.orchestra.run.smithi082.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-09-10T19:33:32.634 INFO:teuthology.orchestra.run.smithi082.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-10T19:33:32.634 INFO:teuthology.orchestra.run.smithi082.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-10T19:33:32.728 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-09-10T19:33:32.739 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:33:32.739 INFO:teuthology.orchestra.run.smithi082.stdout: Package Architecture Version Repository Size 2024-09-10T19:33:32.739 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:33:32.739 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-09-10T19:33:32.739 INFO:teuthology.orchestra.run.smithi082.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-10T19:33:32.739 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-09-10T19:33:32.739 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-10T19:33:32.740 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-10T19:33:32.740 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-10T19:33:32.740 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:33:32.740 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-09-10T19:33:32.740 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:33:32.740 INFO:teuthology.orchestra.run.smithi082.stdout:Install 4 Packages 2024-09-10T19:33:32.740 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:33:32.741 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 79 M 2024-09-10T19:33:32.741 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 126 M 2024-09-10T19:33:32.741 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-09-10T19:33:32.860 INFO:teuthology.orchestra.run.smithi082.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 309 kB/s | 23 kB 00:00 2024-09-10T19:33:33.219 INFO:teuthology.orchestra.run.smithi082.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-09-10T19:33:33.777 INFO:teuthology.orchestra.run.smithi082.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 35 MB/s | 35 MB 00:00 2024-09-10T19:33:34.036 INFO:teuthology.orchestra.run.smithi082.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 24 MB/s | 29 MB 00:01 2024-09-10T19:33:34.036 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:33:34.036 INFO:teuthology.orchestra.run.smithi082.stdout:Total 61 MB/s | 79 MB 00:01 2024-09-10T19:33:34.717 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-09-10T19:33:35.073 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-09-10T19:33:35.073 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-09-10T19:33:36.022 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-09-10T19:33:36.022 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-09-10T19:33:38.451 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-09-10T19:33:39.254 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-10T19:33:39.346 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T19:33:42.123 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T19:33:42.255 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T19:33:47.186 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T19:33:47.305 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T19:33:52.153 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T19:34:30.618 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T19:34:30.659 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-10T19:34:31.032 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T19:34:31.072 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-10T19:34:31.072 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T19:34:31.072 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T19:34:31.864 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T19:34:31.865 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:34:31.865 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-09-10T19:34:31.865 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-10T19:34:31.865 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-10T19:34:31.865 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-10T19:34:31.865 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-10T19:34:31.865 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:34:31.865 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-09-10T19:34:32.207 DEBUG:teuthology.orchestra.run.smithi082:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T19:34:32.286 INFO:teuthology.orchestra.run.smithi082.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T19:34:32.287 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T19:34:32.287 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-10T19:34:32.287 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-10T19:34:32.287 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-10T19:34:32.287 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-10T19:34:32.288 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi082.front.sepia.ceph.com, path=None, version=distro) 2024-09-10T19:34:32.288 DEBUG:teuthology.orchestra.run.smithi082:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T19:34:32.346 INFO:teuthology.orchestra.run.smithi082.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T19:34:32.346 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T19:34:32.346 DEBUG:teuthology.orchestra.run.smithi082:> sudo rpm -qi grub2-tools 2024-09-10T19:34:32.388 INFO:teuthology.orchestra.run.smithi082.stdout:Name : grub2-tools 2024-09-10T19:34:32.388 INFO:teuthology.orchestra.run.smithi082.stdout:Epoch : 1 2024-09-10T19:34:32.388 INFO:teuthology.orchestra.run.smithi082.stdout:Version : 2.06 2024-09-10T19:34:32.388 INFO:teuthology.orchestra.run.smithi082.stdout:Release : 82.el9 2024-09-10T19:34:32.388 INFO:teuthology.orchestra.run.smithi082.stdout:Architecture: x86_64 2024-09-10T19:34:32.388 INFO:teuthology.orchestra.run.smithi082.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-10T19:34:32.388 INFO:teuthology.orchestra.run.smithi082.stdout:Group : Unspecified 2024-09-10T19:34:32.388 INFO:teuthology.orchestra.run.smithi082.stdout:Size : 8274670 2024-09-10T19:34:32.389 INFO:teuthology.orchestra.run.smithi082.stdout:License : GPLv3+ 2024-09-10T19:34:32.389 INFO:teuthology.orchestra.run.smithi082.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-10T19:34:32.389 INFO:teuthology.orchestra.run.smithi082.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-10T19:34:32.389 INFO:teuthology.orchestra.run.smithi082.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-10T19:34:32.389 INFO:teuthology.orchestra.run.smithi082.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-10T19:34:32.389 INFO:teuthology.orchestra.run.smithi082.stdout:Packager : builder@centos.org 2024-09-10T19:34:32.389 INFO:teuthology.orchestra.run.smithi082.stdout:Vendor : CentOS 2024-09-10T19:34:32.389 INFO:teuthology.orchestra.run.smithi082.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-10T19:34:32.389 INFO:teuthology.orchestra.run.smithi082.stdout:Summary : Support tools for GRUB. 2024-09-10T19:34:32.389 INFO:teuthology.orchestra.run.smithi082.stdout:Description : 2024-09-10T19:34:32.389 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:34:32.389 INFO:teuthology.orchestra.run.smithi082.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-10T19:34:32.389 INFO:teuthology.orchestra.run.smithi082.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-10T19:34:32.390 INFO:teuthology.orchestra.run.smithi082.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-10T19:34:32.390 INFO:teuthology.orchestra.run.smithi082.stdout:hardware devices. 2024-09-10T19:34:32.390 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:34:32.390 INFO:teuthology.orchestra.run.smithi082.stdout:This subpackage provides tools for support of all platforms. 2024-09-10T19:34:32.391 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-10T19:34:32.391 INFO:teuthology.task.kernel:Updating grub on smithi082 to boot 5.14.0-505.el9.x86_64 2024-09-10T19:34:32.391 DEBUG:teuthology.orchestra.run.smithi082:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-10T19:34:33.034 INFO:teuthology.orchestra.run.smithi082.stderr:Generating grub configuration file ... 2024-09-10T19:34:34.285 INFO:teuthology.orchestra.run.smithi082.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-10T19:34:34.322 INFO:teuthology.orchestra.run.smithi082.stderr:done 2024-09-10T19:34:34.325 DEBUG:teuthology.orchestra.run.smithi082:> mktemp 2024-09-10T19:34:34.340 INFO:teuthology.orchestra.run.smithi082.stdout:/tmp/tmp.qm9TIUwq3J 2024-09-10T19:34:34.341 DEBUG:teuthology.orchestra.run.smithi082:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.qm9TIUwq3J 2024-09-10T19:34:34.408 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0666 /tmp/tmp.qm9TIUwq3J 2024-09-10T19:34:34.546 DEBUG:teuthology.orchestra.remote:smithi082:/tmp/tmp.qm9TIUwq3J is 6KB 2024-09-10T19:34:34.566 DEBUG:teuthology.orchestra.run.smithi082:> rm -fr /tmp/tmp.qm9TIUwq3J 2024-09-10T19:34:34.580 DEBUG:teuthology.orchestra.run.smithi082:> sudo /bin/ls /boot/loader/entries || true 2024-09-10T19:34:34.644 INFO:teuthology.orchestra.run.smithi082.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-10T19:34:34.644 INFO:teuthology.orchestra.run.smithi082.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-10T19:34:34.644 INFO:teuthology.orchestra.run.smithi082.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-10T19:34:34.644 INFO:teuthology.orchestra.run.smithi082.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-10T19:34:34.645 DEBUG:teuthology.orchestra.run.smithi082:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-10T19:34:34.800 DEBUG:teuthology.orchestra.run.smithi082:> sudo shutdown -r now 2024-09-10T19:35:04.830 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-10T19:35:04.831 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-09-10T19:35:04.832 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T19:35:23.201 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.82 2024-09-10T19:35:32.209 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-09-10T19:35:32.210 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T19:35:35.261 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.82 2024-09-10T19:35:47.273 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-09-10T19:35:47.274 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T19:36:19.093 DEBUG:teuthology.orchestra.run.smithi082:> true 2024-09-10T19:36:19.432 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-09-10T19:36:19.432 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-10T19:36:19.432 DEBUG:teuthology.orchestra.run.smithi082:> uname -r 2024-09-10T19:36:19.447 INFO:teuthology.orchestra.run.smithi082.stdout:5.14.0-505.el9.x86_64 2024-09-10T19:36:19.447 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-10T19:36:19.447 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-10T19:36:19.447 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-10T19:36:20.449 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-10T19:36:20.449 DEBUG:teuthology.orchestra.run.smithi082:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-10T19:36:20.521 INFO:teuthology.orchestra.run.smithi082.stdout:ttyS1 2024-09-10T19:36:20.545 DEBUG:teuthology.parallel:result is None 2024-09-10T19:36:20.545 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-10T19:36:20.558 INFO:teuthology.task.internal:Creating test directory... 2024-09-10T19:36:20.558 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-10T19:36:20.574 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-10T19:36:20.591 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-10T19:36:20.625 INFO:teuthology.task.internal:Creating archive directory... 2024-09-10T19:36:20.625 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-10T19:36:20.669 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-10T19:36:20.676 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-10T19:36:20.676 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-10T19:36:20.743 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T19:36:20.754 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T19:36:20.756 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-10T19:36:20.762 INFO:teuthology.task.internal:Configuring sudo... 2024-09-10T19:36:20.763 DEBUG:teuthology.orchestra.run.smithi082:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-10T19:36:20.822 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-10T19:36:20.831 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-10T19:36:20.832 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-10T19:36:20.878 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T19:36:20.957 DEBUG:teuthology.orchestra.run.smithi082:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T19:36:21.009 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T19:36:21.087 DEBUG:teuthology.orchestra.run.smithi082:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T19:36:21.133 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:36:21.133 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-10T19:36:21.198 DEBUG:teuthology.orchestra.run.smithi082:> sudo service rsyslog restart 2024-09-10T19:36:21.283 INFO:teuthology.orchestra.run.smithi082.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T19:36:21.660 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-10T19:36:21.668 INFO:teuthology.task.internal:Starting timer... 2024-09-10T19:36:21.668 INFO:teuthology.run_tasks:Running task pcp... 2024-09-10T19:36:21.714 INFO:teuthology.run_tasks:Running task selinux... 2024-09-10T19:36:21.745 DEBUG:teuthology.orchestra.run.smithi082:> sudo service auditd rotate 2024-09-10T19:36:21.802 INFO:teuthology.orchestra.run.smithi082.stdout:Rotating logs: 2024-09-10T19:36:21.803 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-10T19:36:21.804 DEBUG:teuthology.orchestra.run.smithi082:> /usr/sbin/getenforce 2024-09-10T19:36:21.846 INFO:teuthology.orchestra.run.smithi082.stdout:Permissive 2024-09-10T19:36:21.846 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi082.front.sepia.ceph.com': 'permissive'} 2024-09-10T19:36:21.847 DEBUG:teuthology.orchestra.run.smithi082:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T19:36:21.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:36:21.913 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-10T19:36:21.913 DEBUG:teuthology.orchestra.run.smithi082:> sudo /usr/sbin/setenforce permissive 2024-09-10T19:36:21.980 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-10T19:36:21.990 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-10T19:36:21.993 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-10T19:36:22.105 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-10T19:36:22.118 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-10T19:36:22.119 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi082.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-10T19:39:59.242 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi082.front.sepia.ceph.com')] 2024-09-10T19:39:59.244 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-09-10T19:39:59.245 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T19:39:59.322 DEBUG:teuthology.orchestra.run.smithi082:> true 2024-09-10T19:39:59.395 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-09-10T19:39:59.395 INFO:teuthology.run_tasks:Running task clock... 2024-09-10T19:39:59.415 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-10T19:39:59.415 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-10T19:39:59.416 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T19:39:59.463 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-10T19:39:59.477 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-10T19:39:59.502 INFO:teuthology.orchestra.run.smithi082.stderr:sudo: ntpd: command not found 2024-09-10T19:39:59.513 INFO:teuthology.orchestra.run.smithi082.stdout:506 Cannot talk to daemon 2024-09-10T19:39:59.527 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-10T19:39:59.541 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-10T19:39:59.596 INFO:teuthology.orchestra.run.smithi082.stderr:bash: line 1: ntpq: command not found 2024-09-10T19:39:59.600 INFO:teuthology.orchestra.run.smithi082.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T19:39:59.600 INFO:teuthology.orchestra.run.smithi082.stdout:=============================================================================== 2024-09-10T19:39:59.600 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T19:39:59.600 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T19:39:59.600 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T19:39:59.600 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T19:39:59.601 INFO:teuthology.run_tasks:Running task install... 2024-09-10T19:39:59.609 DEBUG:teuthology.task.install:project ceph 2024-09-10T19:39:59.609 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'}} 2024-09-10T19:39:59.609 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'} 2024-09-10T19:39:59.610 INFO:teuthology.task.install:Using flavor: default 2024-09-10T19:39:59.615 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-10T19:39:59.615 INFO:teuthology.task.install:extra packages: [] 2024-09-10T19:39:59.616 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46', 'tag': None, 'wait_for_package': False} 2024-09-10T19:39:59.616 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T19:39:59.759 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/ 2024-09-10T19:39:59.759 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4858.gbad1f558 2024-09-10T19:39:59.873 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-10T19:39:59.873 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:39:59.873 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-10T19:39:59.900 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-10T19:39:59.900 DEBUG:teuthology.orchestra.run.smithi082:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-10T19:39:59.965 DEBUG:teuthology.orchestra.run.smithi082:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-10T19:40:00.049 DEBUG:teuthology.orchestra.run.smithi082:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-10T19:40:00.113 INFO:teuthology.orchestra.run.smithi082.stdout:check_obsoletes = 1 2024-09-10T19:40:00.115 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum clean all 2024-09-10T19:40:00.341 INFO:teuthology.orchestra.run.smithi082.stdout:45 files removed 2024-09-10T19:40:00.367 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-radosgw 2024-09-10T19:40:00.905 INFO:teuthology.orchestra.run.smithi082.stdout:ceph packages for x86_64 334 kB/s | 84 kB 00:00 2024-09-10T19:40:01.189 INFO:teuthology.orchestra.run.smithi082.stdout:ceph noarch packages 100 kB/s | 24 kB 00:00 2024-09-10T19:40:01.465 INFO:teuthology.orchestra.run.smithi082.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-09-10T19:40:01.948 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.2 MB 00:00 2024-09-10T19:40:03.776 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - AppStream 29 MB/s | 20 MB 00:00 2024-09-10T19:40:09.601 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 6.5 MB 00:01 2024-09-10T19:40:11.177 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 19 kB 00:00 2024-09-10T19:40:12.106 INFO:teuthology.orchestra.run.smithi082.stdout:Extra Packages for Enterprise Linux 28 MB/s | 23 MB 00:00 2024-09-10T19:40:17.470 INFO:teuthology.orchestra.run.smithi082.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-10T19:40:19.837 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-09-10T19:40:19.847 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:40:19.848 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-09-10T19:40:19.848 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:40:19.848 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-09-10T19:40:19.848 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-radosgw x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 11 M 2024-09-10T19:40:19.848 INFO:teuthology.orchestra.run.smithi082.stdout:Upgrading: 2024-09-10T19:40:19.848 INFO:teuthology.orchestra.run.smithi082.stdout: librados2 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 3.5 M 2024-09-10T19:40:19.848 INFO:teuthology.orchestra.run.smithi082.stdout: librbd1 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 3.2 M 2024-09-10T19:40:19.848 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-09-10T19:40:19.848 INFO:teuthology.orchestra.run.smithi082.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-10T19:40:19.848 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-base x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 5.5 M 2024-09-10T19:40:19.848 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-common x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 22 M 2024-09-10T19:40:19.848 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-selinux x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 25 k 2024-09-10T19:40:19.849 INFO:teuthology.orchestra.run.smithi082.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-10T19:40:19.849 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-10T19:40:19.849 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-10T19:40:19.849 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs2 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 755 k 2024-09-10T19:40:19.849 INFO:teuthology.orchestra.run.smithi082.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-10T19:40:19.849 INFO:teuthology.orchestra.run.smithi082.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-10T19:40:19.849 INFO:teuthology.orchestra.run.smithi082.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-10T19:40:19.849 INFO:teuthology.orchestra.run.smithi082.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-10T19:40:19.849 INFO:teuthology.orchestra.run.smithi082.stdout: libradosstriper1 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 509 k 2024-09-10T19:40:19.850 INFO:teuthology.orchestra.run.smithi082.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-10T19:40:19.850 INFO:teuthology.orchestra.run.smithi082.stdout: librgw2 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 5.5 M 2024-09-10T19:40:19.850 INFO:teuthology.orchestra.run.smithi082.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-10T19:40:19.850 INFO:teuthology.orchestra.run.smithi082.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-10T19:40:19.850 INFO:teuthology.orchestra.run.smithi082.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-10T19:40:19.850 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-argparse x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 45 k 2024-09-10T19:40:19.850 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-common x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 143 k 2024-09-10T19:40:19.850 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cephfs x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 161 k 2024-09-10T19:40:19.850 INFO:teuthology.orchestra.run.smithi082.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-10T19:40:19.850 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rados x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 321 k 2024-09-10T19:40:19.850 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rbd x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 306 k 2024-09-10T19:40:19.850 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rgw x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 99 k 2024-09-10T19:40:19.850 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-10T19:40:19.850 INFO:teuthology.orchestra.run.smithi082.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-10T19:40:19.851 INFO:teuthology.orchestra.run.smithi082.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-10T19:40:19.851 INFO:teuthology.orchestra.run.smithi082.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-10T19:40:19.851 INFO:teuthology.orchestra.run.smithi082.stdout:Installing weak dependencies: 2024-09-10T19:40:19.851 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-10T19:40:19.851 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:40:19.851 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-09-10T19:40:19.851 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:40:19.851 INFO:teuthology.orchestra.run.smithi082.stdout:Install 31 Packages 2024-09-10T19:40:19.851 INFO:teuthology.orchestra.run.smithi082.stdout:Upgrade 2 Packages 2024-09-10T19:40:19.851 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:40:19.852 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 62 M 2024-09-10T19:40:19.852 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-09-10T19:40:20.829 INFO:teuthology.orchestra.run.smithi082.stdout:(1/33): ceph-base-19.3.0-4858.gbad1f558.el9.x86 10 MB/s | 5.5 MB 00:00 2024-09-10T19:40:20.879 INFO:teuthology.orchestra.run.smithi082.stdout:(2/33): ceph-selinux-19.3.0-4858.gbad1f558.el9. 505 kB/s | 25 kB 00:00 2024-09-10T19:40:20.971 INFO:teuthology.orchestra.run.smithi082.stdout:(3/33): libcephfs2-19.3.0-4858.gbad1f558.el9.x8 8.1 MB/s | 755 kB 00:00 2024-09-10T19:40:21.088 INFO:teuthology.orchestra.run.smithi082.stdout:(4/33): ceph-radosgw-19.3.0-4858.gbad1f558.el9. 14 MB/s | 11 MB 00:00 2024-09-10T19:40:21.114 INFO:teuthology.orchestra.run.smithi082.stdout:(5/33): libradosstriper1-19.3.0-4858.gbad1f558. 3.5 MB/s | 509 kB 00:00 2024-09-10T19:40:21.164 INFO:teuthology.orchestra.run.smithi082.stdout:(6/33): python3-ceph-argparse-19.3.0-4858.gbad1 906 kB/s | 45 kB 00:00 2024-09-10T19:40:21.256 INFO:teuthology.orchestra.run.smithi082.stdout:(7/33): python3-ceph-common-19.3.0-4858.gbad1f5 1.5 MB/s | 143 kB 00:00 2024-09-10T19:40:21.373 INFO:teuthology.orchestra.run.smithi082.stdout:(8/33): librgw2-19.3.0-4858.gbad1f558.el9.x86_6 19 MB/s | 5.5 MB 00:00 2024-09-10T19:40:21.398 INFO:teuthology.orchestra.run.smithi082.stdout:(9/33): python3-cephfs-19.3.0-4858.gbad1f558.el 1.1 MB/s | 161 kB 00:00 2024-09-10T19:40:21.423 INFO:teuthology.orchestra.run.smithi082.stdout:(10/33): python3-rados-19.3.0-4858.gbad1f558.el 6.2 MB/s | 321 kB 00:00 2024-09-10T19:40:21.449 INFO:teuthology.orchestra.run.smithi082.stdout:(11/33): python3-rbd-19.3.0-4858.gbad1f558.el9. 5.9 MB/s | 306 kB 00:00 2024-09-10T19:40:21.474 INFO:teuthology.orchestra.run.smithi082.stdout:(12/33): python3-rgw-19.3.0-4858.gbad1f558.el9. 1.9 MB/s | 99 kB 00:00 2024-09-10T19:40:21.674 INFO:teuthology.orchestra.run.smithi082.stdout:(13/33): ceph-common-19.3.0-4858.gbad1f558.el9. 16 MB/s | 22 MB 00:01 2024-09-10T19:40:22.350 INFO:teuthology.orchestra.run.smithi082.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 119 kB/s | 107 kB 00:00 2024-09-10T19:40:22.391 INFO:teuthology.orchestra.run.smithi082.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 186 kB/s | 171 kB 00:00 2024-09-10T19:40:22.425 INFO:teuthology.orchestra.run.smithi082.stdout:(16/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 214 kB/s | 160 kB 00:00 2024-09-10T19:40:22.933 INFO:teuthology.orchestra.run.smithi082.stdout:(17/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 78 kB/s | 45 kB 00:00 2024-09-10T19:40:23.200 INFO:teuthology.orchestra.run.smithi082.stdout:(18/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 377 kB/s | 292 kB 00:00 2024-09-10T19:40:23.250 INFO:teuthology.orchestra.run.smithi082.stdout:(19/33): librdkafka-1.6.1-102.el9.x86_64.rpm 771 kB/s | 662 kB 00:00 2024-09-10T19:40:23.526 INFO:teuthology.orchestra.run.smithi082.stdout:(20/33): python3-prettytable-0.7.2-27.el9.noarc 71 kB/s | 42 kB 00:00 2024-09-10T19:40:23.851 INFO:teuthology.orchestra.run.smithi082.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 60 kB/s | 36 kB 00:00 2024-09-10T19:40:23.901 INFO:teuthology.orchestra.run.smithi082.stdout:(22/33): gperftools-libs-2.9.1-2.el9.x86_64.rpm 6.2 MB/s | 309 kB 00:00 2024-09-10T19:40:23.952 INFO:teuthology.orchestra.run.smithi082.stdout:(23/33): qatlib-24.02.0-1.el9.x86_64.rpm 294 kB/s | 221 kB 00:00 2024-09-10T19:40:24.002 INFO:teuthology.orchestra.run.smithi082.stdout:(24/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 522 kB/s | 26 kB 00:00 2024-09-10T19:40:24.027 INFO:teuthology.orchestra.run.smithi082.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-10T19:40:24.052 INFO:teuthology.orchestra.run.smithi082.stdout:(26/33): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-09-10T19:40:24.111 INFO:teuthology.orchestra.run.smithi082.stdout:(27/33): libarrow-9.0.0-11.el9.x86_64.rpm 21 MB/s | 4.4 MB 00:00 2024-09-10T19:40:24.145 INFO:teuthology.orchestra.run.smithi082.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-10T19:40:24.169 INFO:teuthology.orchestra.run.smithi082.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-09-10T19:40:24.195 INFO:teuthology.orchestra.run.smithi082.stdout:(30/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 71 kB/s | 47 kB 00:00 2024-09-10T19:40:24.237 INFO:teuthology.orchestra.run.smithi082.stdout:(31/33): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-09-10T19:40:24.437 INFO:teuthology.orchestra.run.smithi082.stdout:(32/33): librados2-19.3.0-4858.gbad1f558.el9.x8 13 MB/s | 3.5 MB 00:00 2024-09-10T19:40:24.496 INFO:teuthology.orchestra.run.smithi082.stdout:(33/33): librbd1-19.3.0-4858.gbad1f558.el9.x86_ 11 MB/s | 3.2 MB 00:00 2024-09-10T19:40:24.500 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:40:24.501 INFO:teuthology.orchestra.run.smithi082.stdout:Total 13 MB/s | 62 MB 00:04 2024-09-10T19:40:24.614 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-09-10T19:40:24.651 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-09-10T19:40:24.651 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-09-10T19:40:25.355 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-09-10T19:40:25.355 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-09-10T19:40:26.872 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-09-10T19:40:26.953 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-10T19:40:26.984 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-10T19:40:27.009 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-10T19:40:27.032 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-10T19:40:27.053 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-10T19:40:27.242 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-10T19:40:27.297 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-10T19:40:27.655 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-10T19:40:27.761 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 7/35 2024-09-10T19:40:27.854 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 7/35 2024-09-10T19:40:27.944 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 8/35 2024-09-10T19:40:27.971 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 9/35 2024-09-10T19:40:28.025 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 9/35 2024-09-10T19:40:28.054 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-10T19:40:28.097 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ceph-argparse-2:19.3.0-4858.gbad1f558.el9. 11/35 2024-09-10T19:40:28.168 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 12/35 2024-09-10T19:40:28.198 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_6 13/35 2024-09-10T19:40:28.262 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_6 13/35 2024-09-10T19:40:28.296 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-10T19:40:28.416 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-10T19:40:28.448 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-10T19:40:28.510 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-10T19:40:28.537 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/35 2024-09-10T19:40:28.566 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-10T19:40:28.599 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-10T19:40:28.637 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-10T19:40:28.953 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-10T19:40:29.013 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 23/35 2024-09-10T19:40:29.096 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 23/35 2024-09-10T19:40:29.126 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 24/35 2024-09-10T19:40:29.162 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-10T19:40:29.650 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-10T19:40:29.690 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 27/35 2024-09-10T19:40:29.772 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 27/35 2024-09-10T19:40:29.849 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 28/35 2024-09-10T19:40:29.876 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ceph-common-2:19.3.0-4858.gbad1f558.el9.x8 29/35 2024-09-10T19:40:32.238 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 30/35 2024-09-10T19:40:32.272 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 30/35 2024-09-10T19:40:32.846 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 30/35 2024-09-10T19:40:32.901 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 31/35 2024-09-10T19:40:32.991 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 31/35 2024-09-10T19:40:32.991 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-10T19:40:32.991 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-10T19:40:32.991 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:40:33.018 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 32/35 2024-09-10T19:40:42.976 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 32/35 2024-09-10T19:40:42.977 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /sys 2024-09-10T19:40:42.977 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /proc 2024-09-10T19:40:42.977 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /mnt 2024-09-10T19:40:42.977 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /var/tmp 2024-09-10T19:40:42.977 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /home 2024-09-10T19:40:42.977 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /root 2024-09-10T19:40:42.977 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /tmp 2024-09-10T19:40:42.977 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:40:43.980 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 33/35 2024-09-10T19:40:44.005 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 33/35 2024-09-10T19:40:44.006 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:40:44.006 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-10T19:40:44.006 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T19:40:44.006 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T19:40:44.006 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:40:44.007 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-10T19:40:44.091 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-10T19:40:44.091 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-10T19:40:45.253 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-10T19:40:45.253 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 1/35 2024-09-10T19:40:45.253 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 2/35 2024-09-10T19:40:45.253 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 3/35 2024-09-10T19:40:45.253 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 4/35 2024-09-10T19:40:45.253 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 5/35 2024-09-10T19:40:45.253 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_6 6/35 2024-09-10T19:40:45.253 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 7/35 2024-09-10T19:40:45.253 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4858.gbad1f558.el9. 8/35 2024-09-10T19:40:45.253 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ceph-common-2:19.3.0-4858.gbad1f558.el9.x8 9/35 2024-09-10T19:40:45.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 10/35 2024-09-10T19:40:45.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 11/35 2024-09-10T19:40:45.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 12/35 2024-09-10T19:40:45.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 13/35 2024-09-10T19:40:45.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-10T19:40:45.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-10T19:40:45.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-10T19:40:45.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-10T19:40:45.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-10T19:40:45.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-10T19:40:45.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-10T19:40:45.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-10T19:40:45.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-10T19:40:45.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-10T19:40:45.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 24/35 2024-09-10T19:40:45.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-10T19:40:45.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-10T19:40:45.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-10T19:40:45.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-10T19:40:45.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-10T19:40:45.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-10T19:40:45.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-10T19:40:45.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 32/35 2024-09-10T19:40:45.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-10T19:40:45.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 34/35 2024-09-10T19:40:46.133 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-10T19:40:46.133 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:40:46.133 INFO:teuthology.orchestra.run.smithi082.stdout:Upgraded: 2024-09-10T19:40:46.133 INFO:teuthology.orchestra.run.smithi082.stdout: librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:40:46.133 INFO:teuthology.orchestra.run.smithi082.stdout: librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:40:46.133 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-09-10T19:40:46.133 INFO:teuthology.orchestra.run.smithi082.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-10T19:40:46.133 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:40:46.133 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:40:46.133 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:40:46.134 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:40:46.134 INFO:teuthology.orchestra.run.smithi082.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-10T19:40:46.134 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-10T19:40:46.134 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-10T19:40:46.134 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:40:46.134 INFO:teuthology.orchestra.run.smithi082.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-10T19:40:46.134 INFO:teuthology.orchestra.run.smithi082.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-10T19:40:46.134 INFO:teuthology.orchestra.run.smithi082.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-10T19:40:46.134 INFO:teuthology.orchestra.run.smithi082.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-10T19:40:46.134 INFO:teuthology.orchestra.run.smithi082.stdout: libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:40:46.134 INFO:teuthology.orchestra.run.smithi082.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-10T19:40:46.134 INFO:teuthology.orchestra.run.smithi082.stdout: librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:40:46.134 INFO:teuthology.orchestra.run.smithi082.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-10T19:40:46.135 INFO:teuthology.orchestra.run.smithi082.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-10T19:40:46.135 INFO:teuthology.orchestra.run.smithi082.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-10T19:40:46.135 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-argparse-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:40:46.135 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:40:46.135 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:40:46.135 INFO:teuthology.orchestra.run.smithi082.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-10T19:40:46.135 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:40:46.135 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:40:46.135 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:40:46.135 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-10T19:40:46.135 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-10T19:40:46.135 INFO:teuthology.orchestra.run.smithi082.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-10T19:40:46.135 INFO:teuthology.orchestra.run.smithi082.stdout: re2-1:20211101-20.el9.x86_64 2024-09-10T19:40:46.135 INFO:teuthology.orchestra.run.smithi082.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-10T19:40:46.135 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:40:46.136 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-09-10T19:40:46.520 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-test 2024-09-10T19:40:47.083 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:30 ago on Tue 10 Sep 2024 07:40:17 PM UTC. 2024-09-10T19:40:47.220 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-09-10T19:40:47.220 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:40:47.221 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-09-10T19:40:47.221 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:40:47.221 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-09-10T19:40:47.221 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-test x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 50 M 2024-09-10T19:40:47.221 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-09-10T19:40:47.221 INFO:teuthology.orchestra.run.smithi082.stdout: libcephsqlite x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 161 k 2024-09-10T19:40:47.221 INFO:teuthology.orchestra.run.smithi082.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-10T19:40:47.221 INFO:teuthology.orchestra.run.smithi082.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-10T19:40:47.221 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:40:47.221 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-09-10T19:40:47.221 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:40:47.221 INFO:teuthology.orchestra.run.smithi082.stdout:Install 4 Packages 2024-09-10T19:40:47.222 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:40:47.222 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 50 M 2024-09-10T19:40:47.222 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 215 M 2024-09-10T19:40:47.222 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-09-10T19:40:47.562 INFO:teuthology.orchestra.run.smithi082.stdout:(1/4): libcephsqlite-19.3.0-4858.gbad1f558.el9. 749 kB/s | 161 kB 00:00 2024-09-10T19:40:48.171 INFO:teuthology.orchestra.run.smithi082.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 105 kB/s | 64 kB 00:00 2024-09-10T19:40:48.313 INFO:teuthology.orchestra.run.smithi082.stdout:(3/4): socat-1.7.4.1-6.el9.x86_64.rpm 315 kB/s | 304 kB 00:00 2024-09-10T19:40:48.863 INFO:teuthology.orchestra.run.smithi082.stdout:(4/4): ceph-test-19.3.0-4858.gbad1f558.el9.x86_ 33 MB/s | 50 MB 00:01 2024-09-10T19:40:48.865 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:40:48.866 INFO:teuthology.orchestra.run.smithi082.stdout:Total 31 MB/s | 50 MB 00:01 2024-09-10T19:40:48.893 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-09-10T19:40:48.913 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-09-10T19:40:48.914 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-09-10T19:40:49.293 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-09-10T19:40:49.293 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-09-10T19:40:49.829 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-09-10T19:40:49.907 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-10T19:40:49.945 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-10T19:40:49.973 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 3/4 2024-09-10T19:40:54.694 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 3/4 2024-09-10T19:40:54.771 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 4/4 2024-09-10T19:40:57.087 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 4/4 2024-09-10T19:40:57.088 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 1/4 2024-09-10T19:40:57.088 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 2/4 2024-09-10T19:40:57.088 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-10T19:40:57.384 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-10T19:40:57.384 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:40:57.384 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-09-10T19:40:57.384 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:40:57.384 INFO:teuthology.orchestra.run.smithi082.stdout: libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:40:57.384 INFO:teuthology.orchestra.run.smithi082.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-10T19:40:57.384 INFO:teuthology.orchestra.run.smithi082.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-10T19:40:57.384 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:40:57.385 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-09-10T19:40:57.680 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph 2024-09-10T19:40:58.263 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:41 ago on Tue 10 Sep 2024 07:40:17 PM UTC. 2024-09-10T19:40:58.407 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-09-10T19:40:58.409 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:40:58.409 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-09-10T19:40:58.410 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:40:58.410 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-09-10T19:40:58.410 INFO:teuthology.orchestra.run.smithi082.stdout: ceph x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 6.6 k 2024-09-10T19:40:58.410 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-09-10T19:40:58.410 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mds x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 2.4 M 2024-09-10T19:40:58.410 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 1.5 M 2024-09-10T19:40:58.410 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 291 k 2024-09-10T19:40:58.410 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mon x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 4.8 M 2024-09-10T19:40:58.410 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-osd x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 17 M 2024-09-10T19:40:58.410 INFO:teuthology.orchestra.run.smithi082.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-10T19:40:58.410 INFO:teuthology.orchestra.run.smithi082.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-10T19:40:58.410 INFO:teuthology.orchestra.run.smithi082.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-10T19:40:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-10T19:40:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-10T19:40:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-10T19:40:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-10T19:40:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-10T19:40:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-10T19:40:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-10T19:40:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-collections 2024-09-10T19:40:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-10T19:40:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-10T19:40:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-10T19:40:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-10T19:40:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-10T19:40:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-10T19:40:58.412 INFO:teuthology.orchestra.run.smithi082.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-10T19:40:58.412 INFO:teuthology.orchestra.run.smithi082.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-10T19:40:58.412 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-10T19:40:58.412 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-10T19:40:58.412 INFO:teuthology.orchestra.run.smithi082.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-10T19:40:58.412 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-10T19:40:58.412 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-10T19:40:58.412 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-10T19:40:58.412 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-10T19:40:58.412 INFO:teuthology.orchestra.run.smithi082.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-10T19:40:58.412 INFO:teuthology.orchestra.run.smithi082.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-10T19:40:58.412 INFO:teuthology.orchestra.run.smithi082.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-10T19:40:58.412 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-10T19:40:58.413 INFO:teuthology.orchestra.run.smithi082.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-10T19:40:58.413 INFO:teuthology.orchestra.run.smithi082.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-10T19:40:58.413 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:40:58.413 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-09-10T19:40:58.413 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:40:58.413 INFO:teuthology.orchestra.run.smithi082.stdout:Install 37 Packages 2024-09-10T19:40:58.413 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:40:58.414 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 31 M 2024-09-10T19:40:58.414 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 109 M 2024-09-10T19:40:58.414 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-09-10T19:40:59.164 INFO:teuthology.orchestra.run.smithi082.stdout:(1/37): ceph-19.3.0-4858.gbad1f558.el9.x86_64.r 47 kB/s | 6.6 kB 00:00 2024-09-10T19:40:59.322 INFO:teuthology.orchestra.run.smithi082.stdout:(2/37): ceph-mgr-19.3.0-4858.gbad1f558.el9.x86_ 5.1 MB/s | 1.5 MB 00:00 2024-09-10T19:40:59.372 INFO:teuthology.orchestra.run.smithi082.stdout:(3/37): ceph-mds-19.3.0-4858.gbad1f558.el9.x86_ 6.8 MB/s | 2.4 MB 00:00 2024-09-10T19:40:59.439 INFO:teuthology.orchestra.run.smithi082.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4858.gbad1 4.3 MB/s | 291 kB 00:00 2024-09-10T19:40:59.881 INFO:teuthology.orchestra.run.smithi082.stdout:(5/37): ceph-osd-19.3.0-4858.gbad1f558.el9.x86_ 31 MB/s | 17 MB 00:00 2024-09-10T19:40:59.948 INFO:teuthology.orchestra.run.smithi082.stdout:(6/37): ceph-mon-19.3.0-4858.gbad1f558.el9.x86_ 6.1 MB/s | 4.8 MB 00:00 2024-09-10T19:41:00.424 INFO:teuthology.orchestra.run.smithi082.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 257 kB/s | 253 kB 00:00 2024-09-10T19:41:00.691 INFO:teuthology.orchestra.run.smithi082.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 144 kB/s | 106 kB 00:00 2024-09-10T19:41:01.140 INFO:teuthology.orchestra.run.smithi082.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 189 kB/s | 135 kB 00:00 2024-09-10T19:41:01.416 INFO:teuthology.orchestra.run.smithi082.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 172 kB/s | 125 kB 00:00 2024-09-10T19:41:01.549 INFO:teuthology.orchestra.run.smithi082.stdout:(11/37): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 188 kB 00:00 2024-09-10T19:41:01.641 INFO:teuthology.orchestra.run.smithi082.stdout:(12/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-09-10T19:41:01.717 INFO:teuthology.orchestra.run.smithi082.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 463 kB/s | 35 kB 00:00 2024-09-10T19:41:01.800 INFO:teuthology.orchestra.run.smithi082.stdout:(14/37): python3-packaging-20.9-5.el9.noarch.rp 927 kB/s | 77 kB 00:00 2024-09-10T19:41:01.884 INFO:teuthology.orchestra.run.smithi082.stdout:(15/37): python3-cryptography-36.0.1-4.el9.x86_ 627 kB/s | 1.2 MB 00:02 2024-09-10T19:41:01.909 INFO:teuthology.orchestra.run.smithi082.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 468 kB/s | 51 kB 00:00 2024-09-10T19:41:01.934 INFO:teuthology.orchestra.run.smithi082.stdout:(17/37): python3-urllib3-1.26.5-6.el9.noarch.rp 271 kB/s | 215 kB 00:00 2024-09-10T19:41:01.993 INFO:teuthology.orchestra.run.smithi082.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-09-10T19:41:02.018 INFO:teuthology.orchestra.run.smithi082.stdout:(19/37): lua-devel-5.4.4-4.el9.x86_64.rpm 205 kB/s | 22 kB 00:00 2024-09-10T19:41:02.044 INFO:teuthology.orchestra.run.smithi082.stdout:(20/37): python3-toml-0.10.2-6.el9.noarch.rpm 262 kB/s | 42 kB 00:00 2024-09-10T19:41:02.077 INFO:teuthology.orchestra.run.smithi082.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 518 kB/s | 43 kB 00:00 2024-09-10T19:41:02.111 INFO:teuthology.orchestra.run.smithi082.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 453 kB/s | 11 kB 00:00 2024-09-10T19:41:02.136 INFO:teuthology.orchestra.run.smithi082.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 767 kB/s | 18 kB 00:00 2024-09-10T19:41:02.161 INFO:teuthology.orchestra.run.smithi082.stdout:(24/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-09-10T19:41:02.187 INFO:teuthology.orchestra.run.smithi082.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-09-10T19:41:02.212 INFO:teuthology.orchestra.run.smithi082.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-10T19:41:02.237 INFO:teuthology.orchestra.run.smithi082.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2024-09-10T19:41:02.262 INFO:teuthology.orchestra.run.smithi082.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 926 kB/s | 46 kB 00:00 2024-09-10T19:41:02.288 INFO:teuthology.orchestra.run.smithi082.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-10T19:41:02.313 INFO:teuthology.orchestra.run.smithi082.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 659 kB/s | 16 kB 00:00 2024-09-10T19:41:02.338 INFO:teuthology.orchestra.run.smithi082.stdout:(31/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.2 MB/s | 358 kB 00:00 2024-09-10T19:41:02.364 INFO:teuthology.orchestra.run.smithi082.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-10T19:41:02.389 INFO:teuthology.orchestra.run.smithi082.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 715 kB/s | 36 kB 00:00 2024-09-10T19:41:02.414 INFO:teuthology.orchestra.run.smithi082.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-10T19:41:02.439 INFO:teuthology.orchestra.run.smithi082.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 803 kB/s | 20 kB 00:00 2024-09-10T19:41:02.464 INFO:teuthology.orchestra.run.smithi082.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-10T19:41:02.490 INFO:teuthology.orchestra.run.smithi082.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-10T19:41:02.495 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:41:02.495 INFO:teuthology.orchestra.run.smithi082.stdout:Total 7.5 MB/s | 31 MB 00:04 2024-09-10T19:41:02.646 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-09-10T19:41:02.685 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-09-10T19:41:02.685 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-09-10T19:41:03.156 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-09-10T19:41:03.156 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-09-10T19:41:04.534 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-09-10T19:41:04.601 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-10T19:41:04.634 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-10T19:41:04.662 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-10T19:41:04.732 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-10T19:41:04.786 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-10T19:41:04.819 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-10T19:41:04.910 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-10T19:41:04.997 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-10T19:41:05.038 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-10T19:41:05.077 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-10T19:41:05.103 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-10T19:41:05.137 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-10T19:41:05.171 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-10T19:41:05.219 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-10T19:41:05.254 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-10T19:41:05.297 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-10T19:41:05.346 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-10T19:41:05.378 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-10T19:41:05.451 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-10T19:41:05.588 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-10T19:41:05.681 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-10T19:41:05.734 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-10T19:41:05.773 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-10T19:41:05.821 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-10T19:41:05.877 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-10T19:41:06.068 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-10T19:41:06.118 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-10T19:41:06.196 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-10T19:41:06.285 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-10T19:41:06.335 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-10T19:41:06.462 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-10T19:41:06.624 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4858.gbad1f558.el9. 32/37 2024-09-10T19:41:06.653 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 33/37 2024-09-10T19:41:06.680 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 33/37 2024-09-10T19:41:06.680 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:41:06.680 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-10T19:41:06.680 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T19:41:06.680 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T19:41:06.680 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:08.138 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 34/37 2024-09-10T19:41:08.164 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 34/37 2024-09-10T19:41:08.165 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:41:08.165 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-10T19:41:08.165 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T19:41:08.165 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T19:41:08.165 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:08.621 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 35/37 2024-09-10T19:41:08.646 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 35/37 2024-09-10T19:41:08.646 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:41:08.646 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-10T19:41:08.646 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T19:41:08.646 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T19:41:08.646 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:08.872 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 36/37 2024-09-10T19:41:08.895 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 36/37 2024-09-10T19:41:08.895 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:41:08.895 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-10T19:41:08.895 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T19:41:08.896 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T19:41:08.896 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:08.952 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 37/37 2024-09-10T19:41:10.760 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 37/37 2024-09-10T19:41:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 1/37 2024-09-10T19:41:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 2/37 2024-09-10T19:41:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 3/37 2024-09-10T19:41:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 4/37 2024-09-10T19:41:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 5/37 2024-09-10T19:41:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4858.gbad1f558.el9. 6/37 2024-09-10T19:41:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-10T19:41:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-10T19:41:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-10T19:41:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-10T19:41:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-10T19:41:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-10T19:41:10.765 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-10T19:41:10.765 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-10T19:41:10.765 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-10T19:41:10.765 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-10T19:41:10.765 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-10T19:41:10.766 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-10T19:41:10.766 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-10T19:41:10.766 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-10T19:41:10.766 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-10T19:41:10.766 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-10T19:41:10.766 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-10T19:41:10.766 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-10T19:41:10.766 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-10T19:41:10.766 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-10T19:41:10.766 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-10T19:41:10.766 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-10T19:41:10.766 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-10T19:41:10.766 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-10T19:41:10.766 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-10T19:41:10.767 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-10T19:41:10.767 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-10T19:41:10.767 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-10T19:41:10.767 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-10T19:41:10.767 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-10T19:41:11.885 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-10T19:41:11.885 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:11.885 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-09-10T19:41:11.885 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:41:11.885 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:41:11.885 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:41:11.886 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-modules-core-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T19:41:11.886 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:41:11.886 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:41:11.886 INFO:teuthology.orchestra.run.smithi082.stdout: lua-5.4.4-4.el9.x86_64 2024-09-10T19:41:11.886 INFO:teuthology.orchestra.run.smithi082.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-10T19:41:11.886 INFO:teuthology.orchestra.run.smithi082.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-10T19:41:11.886 INFO:teuthology.orchestra.run.smithi082.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-10T19:41:11.886 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-10T19:41:11.886 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-10T19:41:11.886 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-10T19:41:11.886 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-10T19:41:11.886 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-10T19:41:11.886 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-10T19:41:11.887 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-10T19:41:11.887 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-10T19:41:11.887 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-10T19:41:11.887 INFO:teuthology.orchestra.run.smithi082.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-10T19:41:11.887 INFO:teuthology.orchestra.run.smithi082.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-10T19:41:11.887 INFO:teuthology.orchestra.run.smithi082.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-10T19:41:11.887 INFO:teuthology.orchestra.run.smithi082.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-10T19:41:11.887 INFO:teuthology.orchestra.run.smithi082.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-10T19:41:11.887 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-10T19:41:11.887 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ply-3.11-14.el9.noarch 2024-09-10T19:41:11.887 INFO:teuthology.orchestra.run.smithi082.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-10T19:41:11.887 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-10T19:41:11.887 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-10T19:41:11.887 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-10T19:41:11.888 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-10T19:41:11.888 INFO:teuthology.orchestra.run.smithi082.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-10T19:41:11.888 INFO:teuthology.orchestra.run.smithi082.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-10T19:41:11.888 INFO:teuthology.orchestra.run.smithi082.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-10T19:41:11.888 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-10T19:41:11.888 INFO:teuthology.orchestra.run.smithi082.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-10T19:41:11.888 INFO:teuthology.orchestra.run.smithi082.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-10T19:41:11.888 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:11.888 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-09-10T19:41:12.295 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-base 2024-09-10T19:41:12.865 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:55 ago on Tue 10 Sep 2024 07:40:17 PM UTC. 2024-09-10T19:41:12.954 INFO:teuthology.orchestra.run.smithi082.stdout:Package ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T19:41:12.997 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-09-10T19:41:12.998 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-09-10T19:41:12.998 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-09-10T19:41:13.068 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install cephadm 2024-09-10T19:41:13.629 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:56 ago on Tue 10 Sep 2024 07:40:17 PM UTC. 2024-09-10T19:41:13.760 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-09-10T19:41:13.761 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:41:13.761 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-09-10T19:41:13.761 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:41:13.762 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-09-10T19:41:13.762 INFO:teuthology.orchestra.run.smithi082.stdout: cephadm noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 784 k 2024-09-10T19:41:13.762 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:13.762 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-09-10T19:41:13.762 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:41:13.762 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-09-10T19:41:13.762 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:13.762 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 784 k 2024-09-10T19:41:13.762 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 790 k 2024-09-10T19:41:13.762 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-09-10T19:41:14.040 INFO:teuthology.orchestra.run.smithi082.stdout:cephadm-19.3.0-4858.gbad1f558.el9.noarch.rpm 2.8 MB/s | 784 kB 00:00 2024-09-10T19:41:14.041 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:41:14.041 INFO:teuthology.orchestra.run.smithi082.stdout:Total 2.7 MB/s | 784 kB 00:00 2024-09-10T19:41:14.041 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-09-10T19:41:14.046 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-09-10T19:41:14.047 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-09-10T19:41:14.053 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-09-10T19:41:14.053 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-09-10T19:41:14.123 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-09-10T19:41:14.579 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T19:41:14.671 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T19:41:15.041 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T19:41:15.296 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T19:41:15.296 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:15.296 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-09-10T19:41:15.296 INFO:teuthology.orchestra.run.smithi082.stdout: cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T19:41:15.296 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:15.296 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-09-10T19:41:15.496 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-immutable-object-cache 2024-09-10T19:41:16.065 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:59 ago on Tue 10 Sep 2024 07:40:17 PM UTC. 2024-09-10T19:41:16.198 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-09-10T19:41:16.199 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:41:16.199 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repo Size 2024-09-10T19:41:16.199 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:41:16.199 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-09-10T19:41:16.199 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 145 k 2024-09-10T19:41:16.199 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:16.199 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-09-10T19:41:16.199 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:41:16.199 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-09-10T19:41:16.199 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:16.200 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 145 k 2024-09-10T19:41:16.200 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 432 k 2024-09-10T19:41:16.200 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-09-10T19:41:16.406 INFO:teuthology.orchestra.run.smithi082.stdout:ceph-immutable-object-cache-19.3.0-4858.gbad1f5 703 kB/s | 145 kB 00:00 2024-09-10T19:41:16.407 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:41:16.407 INFO:teuthology.orchestra.run.smithi082.stdout:Total 698 kB/s | 145 kB 00:00 2024-09-10T19:41:16.407 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-09-10T19:41:16.415 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-09-10T19:41:16.415 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-09-10T19:41:16.472 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-09-10T19:41:16.472 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-09-10T19:41:16.641 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-09-10T19:41:16.724 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558. 1/1 2024-09-10T19:41:16.749 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558. 1/1 2024-09-10T19:41:16.749 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:41:16.750 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-10T19:41:16.750 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:17.685 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558. 1/1 2024-09-10T19:41:17.685 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:17.685 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-09-10T19:41:17.685 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:41:17.685 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:17.685 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-09-10T19:41:17.883 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-mgr 2024-09-10T19:41:18.459 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:01 ago on Tue 10 Sep 2024 07:40:17 PM UTC. 2024-09-10T19:41:18.549 INFO:teuthology.orchestra.run.smithi082.stdout:Package ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T19:41:18.591 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-09-10T19:41:18.592 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-09-10T19:41:18.592 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-09-10T19:41:18.667 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-mgr-dashboard 2024-09-10T19:41:19.225 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:02 ago on Tue 10 Sep 2024 07:40:17 PM UTC. 2024-09-10T19:41:19.357 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-09-10T19:41:19.358 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:41:19.358 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-09-10T19:41:19.358 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:41:19.358 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-09-10T19:41:19.359 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 14 M 2024-09-10T19:41:19.359 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-09-10T19:41:19.359 INFO:teuthology.orchestra.run.smithi082.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-10T19:41:19.359 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 34 k 2024-09-10T19:41:19.359 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 16 k 2024-09-10T19:41:19.360 INFO:teuthology.orchestra.run.smithi082.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-10T19:41:19.360 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-10T19:41:19.360 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-10T19:41:19.360 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-10T19:41:19.360 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-10T19:41:19.361 INFO:teuthology.orchestra.run.smithi082.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-10T19:41:19.361 INFO:teuthology.orchestra.run.smithi082.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-10T19:41:19.361 INFO:teuthology.orchestra.run.smithi082.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-10T19:41:19.361 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:19.361 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-09-10T19:41:19.361 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:41:19.362 INFO:teuthology.orchestra.run.smithi082.stdout:Install 12 Packages 2024-09-10T19:41:19.362 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:19.363 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 19 M 2024-09-10T19:41:19.363 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 159 M 2024-09-10T19:41:19.363 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-09-10T19:41:19.643 INFO:teuthology.orchestra.run.smithi082.stdout:(1/12): ceph-prometheus-alerts-19.3.0-4858.gbad 111 kB/s | 16 kB 00:00 2024-09-10T19:41:19.669 INFO:teuthology.orchestra.run.smithi082.stdout:(2/12): ceph-grafana-dashboards-19.3.0-4858.gba 196 kB/s | 34 kB 00:00 2024-09-10T19:41:19.811 INFO:teuthology.orchestra.run.smithi082.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 269 kB 00:00 2024-09-10T19:41:19.861 INFO:teuthology.orchestra.run.smithi082.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2024-09-10T19:41:19.953 INFO:teuthology.orchestra.run.smithi082.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 5.8 MB/s | 548 kB 00:00 2024-09-10T19:41:19.978 INFO:teuthology.orchestra.run.smithi082.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 782 kB/s | 19 kB 00:00 2024-09-10T19:41:20.129 INFO:teuthology.orchestra.run.smithi082.stdout:(7/12): ceph-mgr-dashboard-19.3.0-4858.gbad1f55 22 MB/s | 14 MB 00:00 2024-09-10T19:41:20.204 INFO:teuthology.orchestra.run.smithi082.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 9.1 MB/s | 2.0 MB 00:00 2024-09-10T19:41:20.229 INFO:teuthology.orchestra.run.smithi082.stdout:(9/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.0 MB/s | 863 kB 00:00 2024-09-10T19:41:20.254 INFO:teuthology.orchestra.run.smithi082.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 1.1 MB/s | 144 kB 00:00 2024-09-10T19:41:20.280 INFO:teuthology.orchestra.run.smithi082.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 407 kB/s | 31 kB 00:00 2024-09-10T19:41:20.321 INFO:teuthology.orchestra.run.smithi082.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-09-10T19:41:20.325 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:41:20.326 INFO:teuthology.orchestra.run.smithi082.stdout:Total 20 MB/s | 19 MB 00:00 2024-09-10T19:41:20.372 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-09-10T19:41:20.391 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-09-10T19:41:20.391 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-09-10T19:41:20.667 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-09-10T19:41:20.668 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-09-10T19:41:21.259 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-09-10T19:41:21.338 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-10T19:41:21.374 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-10T19:41:21.419 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-10T19:41:21.451 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-10T19:41:21.554 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-10T19:41:21.684 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-10T19:41:21.771 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-10T19:41:21.814 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-10T19:41:21.851 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-10T19:41:21.883 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4858.gbad1f558.el9 10/12 2024-09-10T19:41:25.063 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4858.gbad1f558.el 11/12 2024-09-10T19:41:25.130 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noa 12/12 2024-09-10T19:41:26.912 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noa 12/12 2024-09-10T19:41:26.912 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4858.gbad1f558.el 1/12 2024-09-10T19:41:26.912 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noa 2/12 2024-09-10T19:41:26.912 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4858.gbad1f558.el9 3/12 2024-09-10T19:41:26.913 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-10T19:41:26.913 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-10T19:41:26.913 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-10T19:41:26.913 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-10T19:41:26.913 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-10T19:41:26.914 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-10T19:41:26.914 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-10T19:41:26.914 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-10T19:41:27.240 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-10T19:41:27.240 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:27.240 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-09-10T19:41:27.241 INFO:teuthology.orchestra.run.smithi082.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-10T19:41:27.241 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-grafana-dashboards-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T19:41:27.241 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T19:41:27.241 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-prometheus-alerts-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T19:41:27.241 INFO:teuthology.orchestra.run.smithi082.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-10T19:41:27.242 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-10T19:41:27.242 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-10T19:41:27.242 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-10T19:41:27.242 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-10T19:41:27.242 INFO:teuthology.orchestra.run.smithi082.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-10T19:41:27.242 INFO:teuthology.orchestra.run.smithi082.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-10T19:41:27.242 INFO:teuthology.orchestra.run.smithi082.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-10T19:41:27.243 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:27.243 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-09-10T19:41:27.497 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-10T19:41:28.077 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:11 ago on Tue 10 Sep 2024 07:40:17 PM UTC. 2024-09-10T19:41:28.219 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-09-10T19:41:28.220 INFO:teuthology.orchestra.run.smithi082.stdout:======================================================================================= 2024-09-10T19:41:28.220 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-09-10T19:41:28.221 INFO:teuthology.orchestra.run.smithi082.stdout:======================================================================================= 2024-09-10T19:41:28.221 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-09-10T19:41:28.221 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 7.4 M 2024-09-10T19:41:28.221 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-09-10T19:41:28.221 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-10T19:41:28.221 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-10T19:41:28.221 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-10T19:41:28.221 INFO:teuthology.orchestra.run.smithi082.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-10T19:41:28.221 INFO:teuthology.orchestra.run.smithi082.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-10T19:41:28.221 INFO:teuthology.orchestra.run.smithi082.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-10T19:41:28.222 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-10T19:41:28.222 INFO:teuthology.orchestra.run.smithi082.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-10T19:41:28.222 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-10T19:41:28.222 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-10T19:41:28.222 INFO:teuthology.orchestra.run.smithi082.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-10T19:41:28.222 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:28.222 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-09-10T19:41:28.222 INFO:teuthology.orchestra.run.smithi082.stdout:======================================================================================= 2024-09-10T19:41:28.222 INFO:teuthology.orchestra.run.smithi082.stdout:Install 12 Packages 2024-09-10T19:41:28.222 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:28.223 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 38 M 2024-09-10T19:41:28.223 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 212 M 2024-09-10T19:41:28.223 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-09-10T19:41:28.462 INFO:teuthology.orchestra.run.smithi082.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-10T19:41:28.504 INFO:teuthology.orchestra.run.smithi082.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.7 MB/s | 801 kB 00:00 2024-09-10T19:41:28.596 INFO:teuthology.orchestra.run.smithi082.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 245 kB/s | 33 kB 00:00 2024-09-10T19:41:28.647 INFO:teuthology.orchestra.run.smithi082.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-09-10T19:41:28.705 INFO:teuthology.orchestra.run.smithi082.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 660 kB/s | 38 kB 00:00 2024-09-10T19:41:28.797 INFO:teuthology.orchestra.run.smithi082.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-48 15 MB/s | 7.4 MB 00:00 2024-09-10T19:41:29.006 INFO:teuthology.orchestra.run.smithi082.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-09-10T19:41:29.115 INFO:teuthology.orchestra.run.smithi082.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.9 MB/s | 3.0 MB 00:00 2024-09-10T19:41:29.173 INFO:teuthology.orchestra.run.smithi082.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.2 MB/s | 250 kB 00:00 2024-09-10T19:41:29.357 INFO:teuthology.orchestra.run.smithi082.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 7.6 MB/s | 4.9 MB 00:00 2024-09-10T19:41:29.599 INFO:teuthology.orchestra.run.smithi082.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 8.6 MB/s | 5.1 MB 00:00 2024-09-10T19:41:30.123 INFO:teuthology.orchestra.run.smithi082.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 17 MB/s | 16 MB 00:00 2024-09-10T19:41:30.124 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:41:30.125 INFO:teuthology.orchestra.run.smithi082.stdout:Total 20 MB/s | 38 MB 00:01 2024-09-10T19:41:30.478 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-09-10T19:41:30.505 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-09-10T19:41:30.505 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-09-10T19:41:30.944 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-09-10T19:41:30.945 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-09-10T19:41:31.707 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-09-10T19:41:31.786 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-10T19:41:31.830 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-10T19:41:31.930 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-10T19:41:31.965 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-10T19:41:32.159 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-10T19:41:32.189 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-10T19:41:32.267 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-10T19:41:32.934 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-10T19:41:33.044 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-10T19:41:34.295 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T19:41:35.327 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-10T19:41:35.355 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f 12/12 2024-09-10T19:41:37.790 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f 12/12 2024-09-10T19:41:37.790 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f 1/12 2024-09-10T19:41:37.790 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-10T19:41:37.790 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-10T19:41:37.790 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-10T19:41:37.790 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-10T19:41:37.790 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-10T19:41:37.790 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-10T19:41:37.791 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-10T19:41:37.791 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-10T19:41:37.791 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T19:41:37.791 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-10T19:41:38.190 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-10T19:41:38.191 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:38.191 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-09-10T19:41:38.191 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T19:41:38.191 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-10T19:41:38.191 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-10T19:41:38.191 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-10T19:41:38.191 INFO:teuthology.orchestra.run.smithi082.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-10T19:41:38.191 INFO:teuthology.orchestra.run.smithi082.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-10T19:41:38.191 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-10T19:41:38.191 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-10T19:41:38.191 INFO:teuthology.orchestra.run.smithi082.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-10T19:41:38.192 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-10T19:41:38.192 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-10T19:41:38.192 INFO:teuthology.orchestra.run.smithi082.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-10T19:41:38.192 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:38.192 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-09-10T19:41:38.591 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-mgr-rook 2024-09-10T19:41:39.173 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:22 ago on Tue 10 Sep 2024 07:40:17 PM UTC. 2024-09-10T19:41:39.306 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-09-10T19:41:39.308 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:41:39.308 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-09-10T19:41:39.308 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:41:39.308 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-09-10T19:41:39.308 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-rook noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 49 k 2024-09-10T19:41:39.308 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-09-10T19:41:39.308 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-10T19:41:39.308 INFO:teuthology.orchestra.run.smithi082.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-10T19:41:39.308 INFO:teuthology.orchestra.run.smithi082.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-10T19:41:39.308 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-10T19:41:39.308 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-10T19:41:39.308 INFO:teuthology.orchestra.run.smithi082.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-10T19:41:39.308 INFO:teuthology.orchestra.run.smithi082.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-10T19:41:39.309 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-10T19:41:39.309 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-10T19:41:39.309 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-10T19:41:39.309 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-10T19:41:39.309 INFO:teuthology.orchestra.run.smithi082.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-10T19:41:39.309 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:39.309 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-09-10T19:41:39.309 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:41:39.309 INFO:teuthology.orchestra.run.smithi082.stdout:Install 13 Packages 2024-09-10T19:41:39.309 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:39.310 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 2.2 M 2024-09-10T19:41:39.310 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 26 M 2024-09-10T19:41:39.310 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-09-10T19:41:39.539 INFO:teuthology.orchestra.run.smithi082.stdout:(1/13): ceph-mgr-rook-19.3.0-4858.gbad1f558.el9 279 kB/s | 49 kB 00:00 2024-09-10T19:41:39.656 INFO:teuthology.orchestra.run.smithi082.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 87 kB/s | 26 kB 00:00 2024-09-10T19:41:39.706 INFO:teuthology.orchestra.run.smithi082.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 56 kB/s | 19 kB 00:00 2024-09-10T19:41:40.082 INFO:teuthology.orchestra.run.smithi082.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 423 kB/s | 159 kB 00:00 2024-09-10T19:41:40.173 INFO:teuthology.orchestra.run.smithi082.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 350 kB/s | 222 kB 00:00 2024-09-10T19:41:40.198 INFO:teuthology.orchestra.run.smithi082.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 459 kB/s | 54 kB 00:00 2024-09-10T19:41:40.249 INFO:teuthology.orchestra.run.smithi082.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 427 kB/s | 32 kB 00:00 2024-09-10T19:41:40.274 INFO:teuthology.orchestra.run.smithi082.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 491 kB/s | 279 kB 00:00 2024-09-10T19:41:40.299 INFO:teuthology.orchestra.run.smithi082.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 141 kB/s | 14 kB 00:00 2024-09-10T19:41:40.333 INFO:teuthology.orchestra.run.smithi082.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-10T19:41:40.375 INFO:teuthology.orchestra.run.smithi082.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 782 kB/s | 59 kB 00:00 2024-09-10T19:41:40.400 INFO:teuthology.orchestra.run.smithi082.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-09-10T19:41:40.434 INFO:teuthology.orchestra.run.smithi082.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.6 MB/s | 1.0 MB 00:00 2024-09-10T19:41:40.436 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:41:40.436 INFO:teuthology.orchestra.run.smithi082.stdout:Total 2.0 MB/s | 2.2 MB 00:01 2024-09-10T19:41:40.501 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-09-10T19:41:40.516 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-09-10T19:41:40.516 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-09-10T19:41:40.604 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-09-10T19:41:40.604 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-09-10T19:41:41.117 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-09-10T19:41:41.287 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-10T19:41:41.343 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-10T19:41:41.398 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-10T19:41:41.427 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-10T19:41:41.469 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-10T19:41:41.569 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-10T19:41:41.671 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-10T19:41:41.728 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-10T19:41:42.433 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-10T19:41:42.485 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-10T19:41:42.518 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-10T19:41:42.566 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-10T19:41:42.588 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 13/13 2024-09-10T19:41:43.290 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 13/13 2024-09-10T19:41:43.290 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 1/13 2024-09-10T19:41:43.290 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-10T19:41:43.291 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-10T19:41:43.291 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-10T19:41:43.291 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-10T19:41:43.291 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-10T19:41:43.291 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-10T19:41:43.291 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-10T19:41:43.291 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-10T19:41:43.291 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-10T19:41:43.291 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-10T19:41:43.291 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-10T19:41:43.640 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-10T19:41:43.640 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:43.640 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-09-10T19:41:43.640 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T19:41:43.640 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-10T19:41:43.641 INFO:teuthology.orchestra.run.smithi082.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-10T19:41:43.641 INFO:teuthology.orchestra.run.smithi082.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-10T19:41:43.641 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-10T19:41:43.641 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-10T19:41:43.641 INFO:teuthology.orchestra.run.smithi082.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-10T19:41:43.641 INFO:teuthology.orchestra.run.smithi082.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-10T19:41:43.641 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-10T19:41:43.641 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-10T19:41:43.641 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-10T19:41:43.641 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-10T19:41:43.641 INFO:teuthology.orchestra.run.smithi082.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-10T19:41:43.641 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:43.641 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-09-10T19:41:43.902 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-mgr-cephadm 2024-09-10T19:41:44.472 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:27 ago on Tue 10 Sep 2024 07:40:17 PM UTC. 2024-09-10T19:41:44.605 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-09-10T19:41:44.606 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:41:44.606 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-09-10T19:41:44.606 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:41:44.606 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-09-10T19:41:44.606 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 154 k 2024-09-10T19:41:44.606 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-09-10T19:41:44.606 INFO:teuthology.orchestra.run.smithi082.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-10T19:41:44.606 INFO:teuthology.orchestra.run.smithi082.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-10T19:41:44.606 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-10T19:41:44.606 INFO:teuthology.orchestra.run.smithi082.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-10T19:41:44.607 INFO:teuthology.orchestra.run.smithi082.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-10T19:41:44.607 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:44.607 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-09-10T19:41:44.607 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:41:44.607 INFO:teuthology.orchestra.run.smithi082.stdout:Install 6 Packages 2024-09-10T19:41:44.607 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:44.607 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 7.0 M 2024-09-10T19:41:44.608 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 33 M 2024-09-10T19:41:44.608 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-09-10T19:41:45.252 INFO:teuthology.orchestra.run.smithi082.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-10T19:41:45.277 INFO:teuthology.orchestra.run.smithi082.stdout:(2/6): ceph-mgr-cephadm-19.3.0-4858.gbad1f558.e 653 kB/s | 154 kB 00:00 2024-09-10T19:41:45.345 INFO:teuthology.orchestra.run.smithi082.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 867 kB/s | 58 kB 00:00 2024-09-10T19:41:45.378 INFO:teuthology.orchestra.run.smithi082.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.2 MB/s | 74 kB 00:00 2024-09-10T19:41:45.420 INFO:teuthology.orchestra.run.smithi082.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.2 MB/s | 548 kB 00:00 2024-09-10T19:41:46.029 INFO:teuthology.orchestra.run.smithi082.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 6.0 MB/s | 6.0 MB 00:00 2024-09-10T19:41:46.032 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:41:46.032 INFO:teuthology.orchestra.run.smithi082.stdout:Total 4.9 MB/s | 7.0 MB 00:01 2024-09-10T19:41:46.105 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-09-10T19:41:46.117 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-09-10T19:41:46.117 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-09-10T19:41:46.217 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-09-10T19:41:46.218 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-09-10T19:41:46.522 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-09-10T19:41:46.693 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-10T19:41:46.747 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-10T19:41:47.268 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-10T19:41:47.358 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-10T19:41:47.435 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-10T19:41:47.468 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 6/6 2024-09-10T19:41:48.217 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 6/6 2024-09-10T19:41:48.217 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/6 2024-09-10T19:41:48.217 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-10T19:41:48.217 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-10T19:41:48.217 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-10T19:41:48.217 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-10T19:41:48.486 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-10T19:41:48.486 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:48.486 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-09-10T19:41:48.486 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T19:41:48.486 INFO:teuthology.orchestra.run.smithi082.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-10T19:41:48.486 INFO:teuthology.orchestra.run.smithi082.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-10T19:41:48.486 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-10T19:41:48.487 INFO:teuthology.orchestra.run.smithi082.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-10T19:41:48.487 INFO:teuthology.orchestra.run.smithi082.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-10T19:41:48.487 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:48.487 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-09-10T19:41:48.733 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-fuse 2024-09-10T19:41:49.295 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:32 ago on Tue 10 Sep 2024 07:40:17 PM UTC. 2024-09-10T19:41:49.425 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-09-10T19:41:49.426 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:41:49.426 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-09-10T19:41:49.426 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:41:49.426 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-09-10T19:41:49.426 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-fuse x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 901 k 2024-09-10T19:41:49.426 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-09-10T19:41:49.426 INFO:teuthology.orchestra.run.smithi082.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-10T19:41:49.426 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:49.426 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-09-10T19:41:49.426 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:41:49.426 INFO:teuthology.orchestra.run.smithi082.stdout:Install 2 Packages 2024-09-10T19:41:49.426 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:49.427 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 981 k 2024-09-10T19:41:49.427 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 2.9 M 2024-09-10T19:41:49.427 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-09-10T19:41:49.655 INFO:teuthology.orchestra.run.smithi082.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 733 kB/s | 80 kB 00:00 2024-09-10T19:41:49.797 INFO:teuthology.orchestra.run.smithi082.stdout:(2/2): ceph-fuse-19.3.0-4858.gbad1f558.el9.x86_ 3.5 MB/s | 901 kB 00:00 2024-09-10T19:41:49.797 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:41:49.798 INFO:teuthology.orchestra.run.smithi082.stdout:Total 2.6 MB/s | 981 kB 00:00 2024-09-10T19:41:49.809 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-09-10T19:41:49.831 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-09-10T19:41:49.831 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-09-10T19:41:49.912 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-09-10T19:41:49.912 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-09-10T19:41:50.097 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-09-10T19:41:50.241 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-10T19:41:50.306 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 2/2 2024-09-10T19:41:50.971 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 2/2 2024-09-10T19:41:50.971 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/2 2024-09-10T19:41:51.261 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-10T19:41:51.261 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:51.261 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-09-10T19:41:51.261 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-10T19:41:51.261 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:51.261 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-09-10T19:41:51.558 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-volume 2024-09-10T19:41:52.141 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:35 ago on Tue 10 Sep 2024 07:40:17 PM UTC. 2024-09-10T19:41:52.278 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-09-10T19:41:52.279 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:41:52.279 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-09-10T19:41:52.279 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:41:52.279 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-09-10T19:41:52.279 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-volume noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 285 k 2024-09-10T19:41:52.279 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:52.279 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-09-10T19:41:52.280 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:41:52.280 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-09-10T19:41:52.280 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:52.280 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 285 k 2024-09-10T19:41:52.280 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 1.3 M 2024-09-10T19:41:52.280 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-09-10T19:41:52.485 INFO:teuthology.orchestra.run.smithi082.stdout:ceph-volume-19.3.0-4858.gbad1f558.el9.noarch.rp 1.4 MB/s | 285 kB 00:00 2024-09-10T19:41:52.486 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:41:52.486 INFO:teuthology.orchestra.run.smithi082.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-09-10T19:41:52.486 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-09-10T19:41:52.495 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-09-10T19:41:52.495 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-09-10T19:41:52.523 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-09-10T19:41:52.523 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-09-10T19:41:52.749 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-09-10T19:41:52.813 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T19:41:52.828 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T19:41:52.829 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:41:52.829 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-10T19:41:52.829 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:53.820 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T19:41:53.820 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:53.821 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-09-10T19:41:53.821 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T19:41:53.821 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:53.821 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-09-10T19:41:54.139 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install librados-devel 2024-09-10T19:41:54.727 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:37 ago on Tue 10 Sep 2024 07:40:17 PM UTC. 2024-09-10T19:41:54.862 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-09-10T19:41:54.863 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:41:54.863 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repo Size 2024-09-10T19:41:54.863 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:41:54.863 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-09-10T19:41:54.863 INFO:teuthology.orchestra.run.smithi082.stdout: librados-devel x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 126 k 2024-09-10T19:41:54.863 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:54.863 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-09-10T19:41:54.863 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:41:54.863 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-09-10T19:41:54.863 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:54.863 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 126 k 2024-09-10T19:41:54.864 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 456 k 2024-09-10T19:41:54.864 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-09-10T19:41:55.082 INFO:teuthology.orchestra.run.smithi082.stdout:librados-devel-19.3.0-4858.gbad1f558.el9.x86_64 580 kB/s | 126 kB 00:00 2024-09-10T19:41:55.083 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:41:55.083 INFO:teuthology.orchestra.run.smithi082.stdout:Total 576 kB/s | 126 kB 00:00 2024-09-10T19:41:55.083 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-09-10T19:41:55.089 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-09-10T19:41:55.089 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-09-10T19:41:55.153 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-09-10T19:41:55.153 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-09-10T19:41:55.303 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-09-10T19:41:55.405 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T19:41:55.830 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T19:41:56.085 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T19:41:56.085 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:56.085 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-09-10T19:41:56.085 INFO:teuthology.orchestra.run.smithi082.stdout: librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:41:56.085 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:56.085 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-09-10T19:41:56.350 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install libcephfs2 2024-09-10T19:41:56.921 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:39 ago on Tue 10 Sep 2024 07:40:17 PM UTC. 2024-09-10T19:41:57.013 INFO:teuthology.orchestra.run.smithi082.stdout:Package libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T19:41:57.057 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-09-10T19:41:57.058 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-09-10T19:41:57.059 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-09-10T19:41:57.130 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install libcephfs-devel 2024-09-10T19:41:57.694 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:40 ago on Tue 10 Sep 2024 07:40:17 PM UTC. 2024-09-10T19:41:57.826 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-09-10T19:41:57.827 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:41:57.827 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repo Size 2024-09-10T19:41:57.827 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:41:57.827 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-09-10T19:41:57.827 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs-devel x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 32 k 2024-09-10T19:41:57.828 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:57.828 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-09-10T19:41:57.828 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:41:57.828 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-09-10T19:41:57.828 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:57.828 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 32 k 2024-09-10T19:41:57.828 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 142 k 2024-09-10T19:41:57.828 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-09-10T19:41:57.969 INFO:teuthology.orchestra.run.smithi082.stdout:libcephfs-devel-19.3.0-4858.gbad1f558.el9.x86_6 225 kB/s | 32 kB 00:00 2024-09-10T19:41:57.970 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:41:57.970 INFO:teuthology.orchestra.run.smithi082.stdout:Total 223 kB/s | 32 kB 00:00 2024-09-10T19:41:57.970 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-09-10T19:41:57.974 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-09-10T19:41:57.974 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-09-10T19:41:57.997 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-09-10T19:41:57.997 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-09-10T19:41:58.085 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-09-10T19:41:58.208 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T19:41:58.570 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T19:41:58.828 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T19:41:58.828 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:58.828 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-09-10T19:41:58.828 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:41:58.828 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:41:58.828 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-09-10T19:41:59.004 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install librados2 2024-09-10T19:41:59.597 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:42 ago on Tue 10 Sep 2024 07:40:17 PM UTC. 2024-09-10T19:41:59.687 INFO:teuthology.orchestra.run.smithi082.stdout:Package librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T19:41:59.730 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-09-10T19:41:59.731 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-09-10T19:41:59.731 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-09-10T19:41:59.799 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install librbd1 2024-09-10T19:42:00.366 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:43 ago on Tue 10 Sep 2024 07:40:17 PM UTC. 2024-09-10T19:42:00.456 INFO:teuthology.orchestra.run.smithi082.stdout:Package librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T19:42:00.499 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-09-10T19:42:00.500 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-09-10T19:42:00.500 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-09-10T19:42:00.570 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install python3-rados 2024-09-10T19:42:01.130 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:44 ago on Tue 10 Sep 2024 07:40:17 PM UTC. 2024-09-10T19:42:01.219 INFO:teuthology.orchestra.run.smithi082.stdout:Package python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T19:42:01.263 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-09-10T19:42:01.264 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-09-10T19:42:01.264 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-09-10T19:42:01.330 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install python3-rgw 2024-09-10T19:42:01.894 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:44 ago on Tue 10 Sep 2024 07:40:17 PM UTC. 2024-09-10T19:42:01.984 INFO:teuthology.orchestra.run.smithi082.stdout:Package python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T19:42:02.026 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-09-10T19:42:02.028 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-09-10T19:42:02.028 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-09-10T19:42:02.099 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install python3-cephfs 2024-09-10T19:42:02.667 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:45 ago on Tue 10 Sep 2024 07:40:17 PM UTC. 2024-09-10T19:42:02.757 INFO:teuthology.orchestra.run.smithi082.stdout:Package python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T19:42:02.801 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-09-10T19:42:02.802 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-09-10T19:42:02.802 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-09-10T19:42:02.870 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install python3-rbd 2024-09-10T19:42:03.473 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:46 ago on Tue 10 Sep 2024 07:40:17 PM UTC. 2024-09-10T19:42:03.564 INFO:teuthology.orchestra.run.smithi082.stdout:Package python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T19:42:03.607 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-09-10T19:42:03.608 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-09-10T19:42:03.608 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-09-10T19:42:03.677 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install rbd-fuse 2024-09-10T19:42:04.246 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:47 ago on Tue 10 Sep 2024 07:40:17 PM UTC. 2024-09-10T19:42:04.379 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-09-10T19:42:04.379 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:42:04.380 INFO:teuthology.orchestra.run.smithi082.stdout: Package Architecture Version Repository Size 2024-09-10T19:42:04.380 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:42:04.380 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-09-10T19:42:04.380 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-fuse x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 85 k 2024-09-10T19:42:04.380 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:04.380 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-09-10T19:42:04.380 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:42:04.380 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-09-10T19:42:04.380 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:04.380 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 85 k 2024-09-10T19:42:04.380 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 231 k 2024-09-10T19:42:04.380 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-09-10T19:42:04.564 INFO:teuthology.orchestra.run.smithi082.stdout:rbd-fuse-19.3.0-4858.gbad1f558.el9.x86_64.rpm 463 kB/s | 85 kB 00:00 2024-09-10T19:42:04.565 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:42:04.565 INFO:teuthology.orchestra.run.smithi082.stdout:Total 459 kB/s | 85 kB 00:00 2024-09-10T19:42:04.565 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-09-10T19:42:04.572 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-09-10T19:42:04.572 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-09-10T19:42:04.638 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-09-10T19:42:04.638 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-09-10T19:42:04.787 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-09-10T19:42:04.886 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T19:42:05.309 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T19:42:05.551 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T19:42:05.551 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:05.551 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-09-10T19:42:05.551 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:42:05.552 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:05.552 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-09-10T19:42:05.806 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install rbd-mirror 2024-09-10T19:42:06.381 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:49 ago on Tue 10 Sep 2024 07:40:17 PM UTC. 2024-09-10T19:42:06.521 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-09-10T19:42:06.521 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:42:06.522 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repo Size 2024-09-10T19:42:06.522 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:42:06.522 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-09-10T19:42:06.522 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-mirror x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 3.2 M 2024-09-10T19:42:06.522 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:06.522 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-09-10T19:42:06.522 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:42:06.522 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-09-10T19:42:06.522 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:06.522 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 3.2 M 2024-09-10T19:42:06.523 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 14 M 2024-09-10T19:42:06.523 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-09-10T19:42:06.843 INFO:teuthology.orchestra.run.smithi082.stdout:rbd-mirror-19.3.0-4858.gbad1f558.el9.x86_64.rpm 9.9 MB/s | 3.2 MB 00:00 2024-09-10T19:42:06.843 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:42:06.844 INFO:teuthology.orchestra.run.smithi082.stdout:Total 9.9 MB/s | 3.2 MB 00:00 2024-09-10T19:42:06.844 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-09-10T19:42:06.853 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-09-10T19:42:06.853 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-09-10T19:42:06.940 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-09-10T19:42:06.940 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-09-10T19:42:07.381 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-09-10T19:42:07.460 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T19:42:07.486 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T19:42:07.486 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:42:07.486 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-10T19:42:07.487 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T19:42:07.487 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T19:42:07.487 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:08.532 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T19:42:08.532 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:08.533 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-09-10T19:42:08.533 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:42:08.533 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:08.533 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-09-10T19:42:08.786 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install rbd-nbd 2024-09-10T19:42:09.380 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:52 ago on Tue 10 Sep 2024 07:40:17 PM UTC. 2024-09-10T19:42:09.516 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-09-10T19:42:09.517 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:42:09.517 INFO:teuthology.orchestra.run.smithi082.stdout: Package Architecture Version Repository Size 2024-09-10T19:42:09.517 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:42:09.517 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-09-10T19:42:09.517 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-nbd x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 170 k 2024-09-10T19:42:09.517 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:09.517 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-09-10T19:42:09.517 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-09-10T19:42:09.518 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-09-10T19:42:09.518 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:09.518 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 170 k 2024-09-10T19:42:09.518 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 490 k 2024-09-10T19:42:09.518 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-09-10T19:42:09.739 INFO:teuthology.orchestra.run.smithi082.stdout:rbd-nbd-19.3.0-4858.gbad1f558.el9.x86_64.rpm 774 kB/s | 170 kB 00:00 2024-09-10T19:42:09.739 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:42:09.740 INFO:teuthology.orchestra.run.smithi082.stdout:Total 767 kB/s | 170 kB 00:00 2024-09-10T19:42:09.740 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-09-10T19:42:09.747 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-09-10T19:42:09.747 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-09-10T19:42:09.811 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-09-10T19:42:09.811 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-09-10T19:42:09.961 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-09-10T19:42:10.061 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T19:42:10.480 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T19:42:10.741 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T19:42:10.741 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:10.741 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-09-10T19:42:10.741 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:42:10.741 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:10.741 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-09-10T19:42:10.967 DEBUG:teuthology.parallel:result is None 2024-09-10T19:42:10.968 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T19:42:11.111 DEBUG:teuthology.orchestra.run.smithi082:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-10T19:42:11.132 INFO:teuthology.orchestra.run.smithi082.stdout:19.3.0-4858.gbad1f558.el9 2024-09-10T19:42:11.133 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4858.gbad1f558.el9 2024-09-10T19:42:11.133 INFO:teuthology.task.install:The correct ceph version 19.3.0-4858.gbad1f558 is installed. 2024-09-10T19:42:11.135 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-10T19:42:11.135 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:11.135 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-10T19:42:11.198 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-10T19:42:11.198 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:11.198 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/daemon-helper 2024-09-10T19:42:11.262 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-10T19:42:11.326 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-10T19:42:11.326 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:11.326 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-10T19:42:11.392 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-10T19:42:11.456 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-10T19:42:11.456 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:11.456 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/stdin-killer 2024-09-10T19:42:11.521 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-10T19:42:11.585 INFO:teuthology.run_tasks:Running task ceph... 2024-09-10T19:42:11.661 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-10T19:42:11.661 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 777 /var/log/ceph 2024-09-10T19:42:11.685 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-10T19:42:11.685 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-10T19:42:11.753 INFO:tasks.ceph:Creating extra log directories... 2024-09-10T19:42:11.753 DEBUG:teuthology.orchestra.run.smithi082:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-10T19:42:11.825 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-10T19:42:11.825 INFO:tasks.ceph:config {'conf': {'osd': {'osd max object name len': 400, 'osd max object namespace len': 64, 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['but it is still running', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-10T19:42:11.825 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899503', 'branch': 'wip-bharath7-testing-2024-09-10-1409', 'description': 'rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{centos_latest}}', 'email': 'skanta@redhat.com', 'first_in_suite': False, 'job_id': '7899503', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-bharath7-testing-2024-09-10-1409'}, 'ceph': {'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'}}, 'workunit': {'branch': 'wip-bharath7-testing-2024-09-10-1409', 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'}}, 'owner': 'scheduled_skanta@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'osd.8', 'osd.9', 'client.0']], 'seed': 5245, 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-bharath7-testing-2024-09-10-1409', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46', 'targets': {'smithi082.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHbr0sYVGCgNZ2RW/V356HQvfWiawJHDCfB9WjUEtOWpfqTyF341DGVdKqOF2ELmlzDkOW6ozg56Rf8gMUeBwh8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'osd max object name len': 400, 'osd max object namespace len': 64, 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'log-ignorelist': ['but it is still running', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'flavor': 'default', 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46', 'cluster': 'ceph'}}, {'workunit': {'clients': {'all': ['rados/test_health_warnings.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-10_15:30:27', 'tube': 'smithi', 'user': 'skanta', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-10T19:42:11.826 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-10T19:42:11.873 DEBUG:teuthology.orchestra.run.smithi082:> sudo install -d -m0777 -- /var/run/ceph 2024-09-10T19:42:11.939 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:11.939 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/scratch_devs of=/dev/stdout 2024-09-10T19:42:11.995 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-10T19:42:11.995 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_1 2024-09-10T19:42:12.050 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-10T19:42:12.050 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T19:42:12.051 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-10T19:42:12.051 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T19:42:12.051 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T19:42:12.051 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-09-10 19:42:07.732147650 +0000 2024-09-10T19:42:12.051 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-09-10 19:39:39.875228638 +0000 2024-09-10T19:42:12.051 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-09-10 19:39:39.875228638 +0000 2024-09-10T19:42:12.051 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-09-10 19:39:39.875228638 +0000 2024-09-10T19:42:12.051 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-10T19:42:12.114 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-09-10T19:42:12.114 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-09-10T19:42:12.114 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000203535 s, 2.5 MB/s 2024-09-10T19:42:12.115 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-10T19:42:12.172 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_2 2024-09-10T19:42:12.227 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-10T19:42:12.227 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T19:42:12.227 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-10T19:42:12.227 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T19:42:12.227 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T19:42:12.228 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-09-10 19:42:07.733147631 +0000 2024-09-10T19:42:12.228 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-09-10 19:39:40.205222558 +0000 2024-09-10T19:42:12.228 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-09-10 19:39:40.205222558 +0000 2024-09-10T19:42:12.228 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-09-10 19:39:40.205222558 +0000 2024-09-10T19:42:12.228 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-10T19:42:12.290 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-09-10T19:42:12.290 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-09-10T19:42:12.290 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000216162 s, 2.4 MB/s 2024-09-10T19:42:12.291 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-10T19:42:12.351 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_3 2024-09-10T19:42:12.411 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-10T19:42:12.411 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T19:42:12.411 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-10T19:42:12.412 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T19:42:12.412 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T19:42:12.412 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-09-10 19:42:07.733147631 +0000 2024-09-10T19:42:12.412 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-09-10 19:39:40.531216551 +0000 2024-09-10T19:42:12.412 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-09-10 19:39:40.531216551 +0000 2024-09-10T19:42:12.412 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-09-10 19:39:40.531216551 +0000 2024-09-10T19:42:12.412 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-10T19:42:12.479 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-09-10T19:42:12.479 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-09-10T19:42:12.479 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000166448 s, 3.1 MB/s 2024-09-10T19:42:12.480 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-10T19:42:12.538 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_4 2024-09-10T19:42:12.595 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-10T19:42:12.596 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T19:42:12.596 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-10T19:42:12.596 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T19:42:12.596 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T19:42:12.596 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-09-10 19:42:07.733147631 +0000 2024-09-10T19:42:12.596 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-09-10 19:39:40.847210729 +0000 2024-09-10T19:42:12.596 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-09-10 19:39:40.847210729 +0000 2024-09-10T19:42:12.596 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-09-10 19:39:40.847210729 +0000 2024-09-10T19:42:12.596 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-10T19:42:12.660 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-09-10T19:42:12.660 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-09-10T19:42:12.660 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.00029597 s, 1.7 MB/s 2024-09-10T19:42:12.661 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-10T19:42:12.717 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-10T19:42:12.717 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi082.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-09-10T19:42:12.717 INFO:tasks.ceph:Generating config... 2024-09-10T19:42:12.719 INFO:tasks.ceph:[osd] osd max object name len = 400 2024-09-10T19:42:12.719 INFO:tasks.ceph:[osd] osd max object namespace len = 64 2024-09-10T19:42:12.719 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-10T19:42:12.719 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-10T19:42:12.719 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-10T19:42:12.719 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-10T19:42:12.719 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-10T19:42:12.719 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-10T19:42:12.719 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-10T19:42:12.719 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-10T19:42:12.719 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-10T19:42:12.719 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-10T19:42:12.719 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-10T19:42:12.720 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-10T19:42:12.720 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-10T19:42:12.720 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-10T19:42:12.720 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-10T19:42:12.720 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-10T19:42:12.720 INFO:tasks.ceph:Setting up mon.a... 2024-09-10T19:42:12.720 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-10T19:42:12.802 INFO:teuthology.orchestra.run.smithi082.stdout:creating /etc/ceph/ceph.keyring 2024-09-10T19:42:12.804 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-10T19:42:12.847 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-10T19:42:12.913 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.82')] 2024-09-10T19:42:12.914 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.82', 'mon election default strategy': 1}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'osd max object name len': 400, 'osd max object namespace len': 64, 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2024-09-10T19:42:12.914 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:12.914 DEBUG:teuthology.orchestra.run.smithi082:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-10T19:42:12.969 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.82 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-10T19:42:13.050 INFO:teuthology.orchestra.run.smithi082.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-10T19:42:13.050 INFO:teuthology.orchestra.run.smithi082.stdout:monmaptool: generated fsid 2964cb92-693e-4d63-a589-7427632fbccd 2024-09-10T19:42:13.050 INFO:teuthology.orchestra.run.smithi082.stdout:setting min_mon_release = quincy 2024-09-10T19:42:13.050 INFO:teuthology.orchestra.run.smithi082.stdout:epoch 0 2024-09-10T19:42:13.050 INFO:teuthology.orchestra.run.smithi082.stdout:fsid 2964cb92-693e-4d63-a589-7427632fbccd 2024-09-10T19:42:13.050 INFO:teuthology.orchestra.run.smithi082.stdout:last_changed 2024-09-10T19:42:13.048884+0000 2024-09-10T19:42:13.050 INFO:teuthology.orchestra.run.smithi082.stdout:created 2024-09-10T19:42:13.048884+0000 2024-09-10T19:42:13.050 INFO:teuthology.orchestra.run.smithi082.stdout:min_mon_release 17 (quincy) 2024-09-10T19:42:13.050 INFO:teuthology.orchestra.run.smithi082.stdout:election_strategy: 1 2024-09-10T19:42:13.050 INFO:teuthology.orchestra.run.smithi082.stdout:0: [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] mon.a 2024-09-10T19:42:13.050 INFO:teuthology.orchestra.run.smithi082.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-09-10T19:42:13.052 DEBUG:teuthology.orchestra.run.smithi082:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-10T19:42:13.107 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 2964cb92-693e-4d63-a589-7427632fbccd... 2024-09-10T19:42:13.108 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-10T19:42:13.190 INFO:teuthology.orchestra.run.smithi082.stdout:[global] 2024-09-10T19:42:13.190 INFO:teuthology.orchestra.run.smithi082.stdout: chdir = "" 2024-09-10T19:42:13.190 INFO:teuthology.orchestra.run.smithi082.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-10T19:42:13.190 INFO:teuthology.orchestra.run.smithi082.stdout: auth supported = cephx 2024-09-10T19:42:13.190 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:13.190 INFO:teuthology.orchestra.run.smithi082.stdout: filestore xattr use omap = true 2024-09-10T19:42:13.190 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:13.190 INFO:teuthology.orchestra.run.smithi082.stdout: mon clock drift allowed = 1.000 2024-09-10T19:42:13.190 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:13.190 INFO:teuthology.orchestra.run.smithi082.stdout: osd crush chooseleaf type = 0 2024-09-10T19:42:13.190 INFO:teuthology.orchestra.run.smithi082.stdout: auth debug = true 2024-09-10T19:42:13.191 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:13.191 INFO:teuthology.orchestra.run.smithi082.stdout: ms die on old message = true 2024-09-10T19:42:13.191 INFO:teuthology.orchestra.run.smithi082.stdout: ms die on bug = true 2024-09-10T19:42:13.191 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:13.191 INFO:teuthology.orchestra.run.smithi082.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-10T19:42:13.191 INFO:teuthology.orchestra.run.smithi082.stdout: mon pg warn max object skew = 0 2024-09-10T19:42:13.191 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:13.191 INFO:teuthology.orchestra.run.smithi082.stdout: # disable pg_autoscaler by default for new pools 2024-09-10T19:42:13.191 INFO:teuthology.orchestra.run.smithi082.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-10T19:42:13.191 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:13.191 INFO:teuthology.orchestra.run.smithi082.stdout: osd pool default size = 2 2024-09-10T19:42:13.191 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:13.191 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd allow primary affinity = true 2024-09-10T19:42:13.192 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd allow pg remap = true 2024-09-10T19:42:13.192 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on legacy crush tunables = false 2024-09-10T19:42:13.192 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on crush straw calc version zero = false 2024-09-10T19:42:13.192 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on no sortbitwise = false 2024-09-10T19:42:13.192 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on osd down out interval zero = false 2024-09-10T19:42:13.192 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on too few osds = false 2024-09-10T19:42:13.192 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-10T19:42:13.192 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-10T19:42:13.192 INFO:teuthology.orchestra.run.smithi082.stdout: mon_allow_pool_size_one = true 2024-09-10T19:42:13.192 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:13.192 INFO:teuthology.orchestra.run.smithi082.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-10T19:42:13.192 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:13.192 INFO:teuthology.orchestra.run.smithi082.stdout: osd default data pool replay window = 5 2024-09-10T19:42:13.193 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:13.193 INFO:teuthology.orchestra.run.smithi082.stdout: mon allow pool delete = true 2024-09-10T19:42:13.193 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:13.193 INFO:teuthology.orchestra.run.smithi082.stdout: mon cluster log file level = debug 2024-09-10T19:42:13.193 INFO:teuthology.orchestra.run.smithi082.stdout: debug asserts on shutdown = true 2024-09-10T19:42:13.193 INFO:teuthology.orchestra.run.smithi082.stdout: mon health detail to clog = false 2024-09-10T19:42:13.193 INFO:teuthology.orchestra.run.smithi082.stdout: mon host = 172.21.15.82 2024-09-10T19:42:13.193 INFO:teuthology.orchestra.run.smithi082.stdout: mon election default strategy = 1 2024-09-10T19:42:13.193 INFO:teuthology.orchestra.run.smithi082.stdout: fsid = 2964cb92-693e-4d63-a589-7427632fbccd 2024-09-10T19:42:13.193 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:13.193 INFO:teuthology.orchestra.run.smithi082.stdout:[osd] 2024-09-10T19:42:13.193 INFO:teuthology.orchestra.run.smithi082.stdout: osd journal size = 100 2024-09-10T19:42:13.193 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:13.193 INFO:teuthology.orchestra.run.smithi082.stdout: osd scrub load threshold = 5.0 2024-09-10T19:42:13.194 INFO:teuthology.orchestra.run.smithi082.stdout: osd scrub max interval = 600 2024-09-10T19:42:13.194 INFO:teuthology.orchestra.run.smithi082.stdout: osd mclock profile = high_recovery_ops 2024-09-10T19:42:13.194 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:13.194 INFO:teuthology.orchestra.run.smithi082.stdout: osd recover clone overlap = true 2024-09-10T19:42:13.194 INFO:teuthology.orchestra.run.smithi082.stdout: osd recovery max chunk = 1048576 2024-09-10T19:42:13.194 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:13.194 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug shutdown = true 2024-09-10T19:42:13.194 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug op order = true 2024-09-10T19:42:13.194 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug verify stray on activate = true 2024-09-10T19:42:13.194 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug trim objects = true 2024-09-10T19:42:13.194 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:13.194 INFO:teuthology.orchestra.run.smithi082.stdout: osd open classes on start = true 2024-09-10T19:42:13.194 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug pg log writeout = true 2024-09-10T19:42:13.194 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:13.195 INFO:teuthology.orchestra.run.smithi082.stdout: osd deep scrub update digest min age = 30 2024-09-10T19:42:13.195 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:13.195 INFO:teuthology.orchestra.run.smithi082.stdout: osd map max advance = 10 2024-09-10T19:42:13.195 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:13.195 INFO:teuthology.orchestra.run.smithi082.stdout: journal zero on create = true 2024-09-10T19:42:13.195 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:13.195 INFO:teuthology.orchestra.run.smithi082.stdout: filestore ondisk finisher threads = 3 2024-09-10T19:42:13.195 INFO:teuthology.orchestra.run.smithi082.stdout: filestore apply finisher threads = 3 2024-09-10T19:42:13.195 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:13.195 INFO:teuthology.orchestra.run.smithi082.stdout: bdev debug aio = true 2024-09-10T19:42:13.195 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug misdirected ops = true 2024-09-10T19:42:13.195 INFO:teuthology.orchestra.run.smithi082.stdout: osd max object name len = 400 2024-09-10T19:42:13.195 INFO:teuthology.orchestra.run.smithi082.stdout: osd max object namespace len = 64 2024-09-10T19:42:13.195 INFO:teuthology.orchestra.run.smithi082.stdout: bluestore zero block detection = True 2024-09-10T19:42:13.195 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2024-09-10T19:42:13.196 INFO:teuthology.orchestra.run.smithi082.stdout: debug osd = 20 2024-09-10T19:42:13.196 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug verify cached snaps = True 2024-09-10T19:42:13.196 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug verify missing on start = True 2024-09-10T19:42:13.196 INFO:teuthology.orchestra.run.smithi082.stdout: osd mclock override recovery settings = True 2024-09-10T19:42:13.196 INFO:teuthology.orchestra.run.smithi082.stdout: osd op queue = debug_random 2024-09-10T19:42:13.196 INFO:teuthology.orchestra.run.smithi082.stdout: osd op queue cut off = debug_random 2024-09-10T19:42:13.196 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:13.196 INFO:teuthology.orchestra.run.smithi082.stdout:[mgr] 2024-09-10T19:42:13.196 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2024-09-10T19:42:13.196 INFO:teuthology.orchestra.run.smithi082.stdout: debug mgr = 20 2024-09-10T19:42:13.196 INFO:teuthology.orchestra.run.smithi082.stdout: debug mon = 20 2024-09-10T19:42:13.196 INFO:teuthology.orchestra.run.smithi082.stdout: debug auth = 20 2024-09-10T19:42:13.196 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min pgs per osd = 4 2024-09-10T19:42:13.196 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min bytes per osd = 10 2024-09-10T19:42:13.197 INFO:teuthology.orchestra.run.smithi082.stdout: mgr/telemetry/nag = false 2024-09-10T19:42:13.197 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:13.197 INFO:teuthology.orchestra.run.smithi082.stdout:[mon] 2024-09-10T19:42:13.197 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2024-09-10T19:42:13.197 INFO:teuthology.orchestra.run.smithi082.stdout: debug mon = 20 2024-09-10T19:42:13.197 INFO:teuthology.orchestra.run.smithi082.stdout: debug paxos = 20 2024-09-10T19:42:13.197 INFO:teuthology.orchestra.run.smithi082.stdout: debug auth = 20 2024-09-10T19:42:13.197 INFO:teuthology.orchestra.run.smithi082.stdout: mon data avail warn = 5 2024-09-10T19:42:13.197 INFO:teuthology.orchestra.run.smithi082.stdout: mon mgr mkfs grace = 240 2024-09-10T19:42:13.197 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min pgs per osd = 4 2024-09-10T19:42:13.197 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd reporter subtree level = osd 2024-09-10T19:42:13.197 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd prime pg temp = true 2024-09-10T19:42:13.197 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min bytes per osd = 10 2024-09-10T19:42:13.198 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:13.198 INFO:teuthology.orchestra.run.smithi082.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-10T19:42:13.198 INFO:teuthology.orchestra.run.smithi082.stdout: auth mon ticket ttl = 660 # 11m 2024-09-10T19:42:13.198 INFO:teuthology.orchestra.run.smithi082.stdout: auth service ticket ttl = 240 # 4m 2024-09-10T19:42:13.198 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:13.198 INFO:teuthology.orchestra.run.smithi082.stdout: # don't complain about insecure global_id in the test suite 2024-09-10T19:42:13.198 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-10T19:42:13.198 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-10T19:42:13.198 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:13.198 INFO:teuthology.orchestra.run.smithi082.stdout: # 1m isn't quite enough 2024-09-10T19:42:13.198 INFO:teuthology.orchestra.run.smithi082.stdout: mon_down_mkfs_grace = 2m 2024-09-10T19:42:13.198 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:13.198 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_filestore_osds = false 2024-09-10T19:42:13.199 INFO:teuthology.orchestra.run.smithi082.stdout: mon scrub interval = 300 2024-09-10T19:42:13.199 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:13.199 INFO:teuthology.orchestra.run.smithi082.stdout:[client] 2024-09-10T19:42:13.199 INFO:teuthology.orchestra.run.smithi082.stdout: rgw cache enabled = true 2024-09-10T19:42:13.199 INFO:teuthology.orchestra.run.smithi082.stdout: rgw enable ops log = true 2024-09-10T19:42:13.199 INFO:teuthology.orchestra.run.smithi082.stdout: rgw enable usage log = true 2024-09-10T19:42:13.199 INFO:teuthology.orchestra.run.smithi082.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-10T19:42:13.199 INFO:teuthology.orchestra.run.smithi082.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-10T19:42:13.199 INFO:teuthology.orchestra.run.smithi082.stdout:[mon.a] 2024-09-10T19:42:13.201 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-10T19:42:13.202 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-10T19:42:13.296 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-10T19:42:13.296 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:13.296 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-10T19:42:13.312 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:13.312 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-10T19:42:13.367 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi082.front.sepia.ceph.com 2024-09-10T19:42:13.368 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:13.368 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-10T19:42:13.368 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-10T19:42:13.442 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:13.443 DEBUG:teuthology.orchestra.run.smithi082:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-10T19:42:13.499 INFO:tasks.ceph:Setting up mon nodes... 2024-09-10T19:42:13.499 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-10T19:42:13.499 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-10T19:42:13.590 INFO:teuthology.orchestra.run.smithi082.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-10T19:42:13.593 INFO:tasks.ceph:Setting up mds nodes... 2024-09-10T19:42:13.593 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-10T19:42:13.593 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-10T19:42:13.633 INFO:teuthology.orchestra.run.smithi082.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-10T19:42:13.646 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-10T19:42:13.646 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi082.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-09-10T19:42:13.646 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-10T19:42:13.711 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-10T19:42:13.711 INFO:tasks.ceph:role: osd.0 2024-09-10T19:42:13.711 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi082.front.sepia.ceph.com 2024-09-10T19:42:13.711 DEBUG:teuthology.orchestra.run.smithi082:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-10T19:42:13.783 INFO:teuthology.orchestra.run.smithi082.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-10T19:42:13.783 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-10T19:42:13.783 INFO:teuthology.orchestra.run.smithi082.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-10T19:42:13.783 INFO:teuthology.orchestra.run.smithi082.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-10T19:42:13.783 INFO:teuthology.orchestra.run.smithi082.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-10T19:42:13.783 INFO:teuthology.orchestra.run.smithi082.stdout: = sunit=0 swidth=0 blks 2024-09-10T19:42:13.783 INFO:teuthology.orchestra.run.smithi082.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-10T19:42:13.783 INFO:teuthology.orchestra.run.smithi082.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-10T19:42:13.783 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-10T19:42:13.783 INFO:teuthology.orchestra.run.smithi082.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-10T19:42:14.260 INFO:teuthology.orchestra.run.smithi082.stdout:Discarding blocks...Done. 2024-09-10T19:42:14.262 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi082.front.sepia.ceph.com -o noatime 2024-09-10T19:42:14.262 DEBUG:teuthology.orchestra.run.smithi082:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-10T19:42:14.394 DEBUG:teuthology.orchestra.run.smithi082:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-10T19:42:14.429 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-10T19:42:14.490 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-10T19:42:14.491 INFO:tasks.ceph:role: osd.1 2024-09-10T19:42:14.491 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi082.front.sepia.ceph.com 2024-09-10T19:42:14.491 DEBUG:teuthology.orchestra.run.smithi082:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-10T19:42:14.563 INFO:teuthology.orchestra.run.smithi082.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-10T19:42:14.563 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-10T19:42:14.563 INFO:teuthology.orchestra.run.smithi082.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-10T19:42:14.563 INFO:teuthology.orchestra.run.smithi082.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-10T19:42:14.563 INFO:teuthology.orchestra.run.smithi082.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-10T19:42:14.564 INFO:teuthology.orchestra.run.smithi082.stdout: = sunit=0 swidth=0 blks 2024-09-10T19:42:14.564 INFO:teuthology.orchestra.run.smithi082.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-10T19:42:14.564 INFO:teuthology.orchestra.run.smithi082.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-10T19:42:14.564 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-10T19:42:14.564 INFO:teuthology.orchestra.run.smithi082.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-10T19:42:15.024 INFO:teuthology.orchestra.run.smithi082.stdout:Discarding blocks...Done. 2024-09-10T19:42:15.026 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi082.front.sepia.ceph.com -o noatime 2024-09-10T19:42:15.026 DEBUG:teuthology.orchestra.run.smithi082:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-10T19:42:15.098 DEBUG:teuthology.orchestra.run.smithi082:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-10T19:42:15.168 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-10T19:42:15.232 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-10T19:42:15.233 INFO:tasks.ceph:role: osd.2 2024-09-10T19:42:15.233 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi082.front.sepia.ceph.com 2024-09-10T19:42:15.233 DEBUG:teuthology.orchestra.run.smithi082:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-10T19:42:15.305 INFO:teuthology.orchestra.run.smithi082.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-10T19:42:15.306 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-10T19:42:15.306 INFO:teuthology.orchestra.run.smithi082.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-10T19:42:15.306 INFO:teuthology.orchestra.run.smithi082.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-10T19:42:15.306 INFO:teuthology.orchestra.run.smithi082.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-10T19:42:15.306 INFO:teuthology.orchestra.run.smithi082.stdout: = sunit=0 swidth=0 blks 2024-09-10T19:42:15.306 INFO:teuthology.orchestra.run.smithi082.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-10T19:42:15.306 INFO:teuthology.orchestra.run.smithi082.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-10T19:42:15.306 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-10T19:42:15.306 INFO:teuthology.orchestra.run.smithi082.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-10T19:42:15.780 INFO:teuthology.orchestra.run.smithi082.stdout:Discarding blocks...Done. 2024-09-10T19:42:15.782 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi082.front.sepia.ceph.com -o noatime 2024-09-10T19:42:15.782 DEBUG:teuthology.orchestra.run.smithi082:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-10T19:42:15.877 DEBUG:teuthology.orchestra.run.smithi082:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-10T19:42:15.908 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-10T19:42:15.973 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-10T19:42:15.973 INFO:tasks.ceph:role: osd.3 2024-09-10T19:42:15.973 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi082.front.sepia.ceph.com 2024-09-10T19:42:15.973 DEBUG:teuthology.orchestra.run.smithi082:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-10T19:42:16.047 INFO:teuthology.orchestra.run.smithi082.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-10T19:42:16.047 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-10T19:42:16.047 INFO:teuthology.orchestra.run.smithi082.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-10T19:42:16.047 INFO:teuthology.orchestra.run.smithi082.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-10T19:42:16.047 INFO:teuthology.orchestra.run.smithi082.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-10T19:42:16.047 INFO:teuthology.orchestra.run.smithi082.stdout: = sunit=0 swidth=0 blks 2024-09-10T19:42:16.047 INFO:teuthology.orchestra.run.smithi082.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-10T19:42:16.047 INFO:teuthology.orchestra.run.smithi082.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-10T19:42:16.047 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-10T19:42:16.047 INFO:teuthology.orchestra.run.smithi082.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-10T19:42:16.561 INFO:teuthology.orchestra.run.smithi082.stdout:Discarding blocks...Done. 2024-09-10T19:42:16.563 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi082.front.sepia.ceph.com -o noatime 2024-09-10T19:42:16.564 DEBUG:teuthology.orchestra.run.smithi082:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-10T19:42:16.704 DEBUG:teuthology.orchestra.run.smithi082:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-10T19:42:16.735 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-10T19:42:16.800 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-10T19:42:16.801 INFO:tasks.ceph:role: osd.4 2024-09-10T19:42:16.801 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-10T19:42:16.866 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-10T19:42:16.866 INFO:tasks.ceph:role: osd.5 2024-09-10T19:42:16.866 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-10T19:42:16.931 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-10T19:42:16.931 INFO:tasks.ceph:role: osd.6 2024-09-10T19:42:16.931 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-10T19:42:16.996 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-10T19:42:16.996 INFO:tasks.ceph:role: osd.7 2024-09-10T19:42:16.996 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2024-09-10T19:42:17.060 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-10T19:42:17.060 INFO:tasks.ceph:role: osd.8 2024-09-10T19:42:17.060 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2024-09-10T19:42:17.125 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-10T19:42:17.126 INFO:tasks.ceph:role: osd.9 2024-09-10T19:42:17.126 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T19:42:17.221 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:17.218+0000 7fa5fbd1d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-10T19:42:17.221 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:17.219+0000 7fa5fbd1d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-10T19:42:17.221 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:17.219+0000 7fa5fbd1d640 -1 bdev(0x559c01afd800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-10T19:42:17.221 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:17.219+0000 7fa5fbd1d640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-10T19:42:19.346 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-10T19:42:19.375 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T19:42:19.464 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:19.461+0000 7fb09500d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-10T19:42:19.464 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:19.462+0000 7fb09500d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-10T19:42:19.464 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:19.462+0000 7fb09500d640 -1 bdev(0x564bd1a09800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-10T19:42:19.464 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:19.462+0000 7fb09500d640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-10T19:42:21.840 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-10T19:42:21.870 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T19:42:21.960 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:21.958+0000 7fdfd7033640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-10T19:42:21.960 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:21.958+0000 7fdfd7033640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-10T19:42:21.960 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:21.958+0000 7fdfd7033640 -1 bdev(0x5643047ed800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-10T19:42:21.960 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:21.958+0000 7fdfd7033640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-10T19:42:24.103 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-10T19:42:24.132 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T19:42:24.223 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:24.221+0000 7f702da0f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-10T19:42:24.223 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:24.221+0000 7f702da0f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-10T19:42:24.223 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:24.221+0000 7f702da0f640 -1 bdev(0x5557393c1800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-10T19:42:24.223 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:24.221+0000 7f702da0f640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-10T19:42:26.349 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-10T19:42:26.377 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T19:42:26.465 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:26.462+0000 7f3e09bae640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-10T19:42:26.465 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:26.463+0000 7f3e09bae640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-10T19:42:26.465 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:26.463+0000 7f3e09bae640 -1 bdev(0x56538b6c7800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-10T19:42:26.465 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:26.463+0000 7f3e09bae640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-10T19:42:28.831 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-10T19:42:28.860 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T19:42:28.952 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:28.950+0000 7f0a9562c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-10T19:42:28.952 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:28.950+0000 7f0a9562c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-10T19:42:28.952 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:28.950+0000 7f0a9562c640 -1 bdev(0x559cf424d800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-10T19:42:28.952 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:28.950+0000 7f0a9562c640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-10T19:42:31.077 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-10T19:42:31.107 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T19:42:31.196 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:31.194+0000 7fd2f3e14640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-10T19:42:31.196 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:31.194+0000 7fd2f3e14640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-10T19:42:31.197 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:31.194+0000 7fd2f3e14640 -1 bdev(0x5633364a1800 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2024-09-10T19:42:31.197 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:31.194+0000 7fd2f3e14640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-10T19:42:33.330 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-10T19:42:33.360 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T19:42:33.451 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:33.448+0000 7f26af433640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-10T19:42:33.451 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:33.448+0000 7f26af433640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-10T19:42:33.451 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:33.448+0000 7f26af433640 -1 bdev(0x56441d581800 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2024-09-10T19:42:33.451 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:33.449+0000 7f26af433640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-10T19:42:35.824 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-10T19:42:35.853 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T19:42:35.946 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:35.943+0000 7fdc2452e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2024-09-10T19:42:35.946 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:35.943+0000 7fdc2452e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2024-09-10T19:42:35.946 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:35.943+0000 7fdc2452e640 -1 bdev(0x5591b0659800 /var/lib/ceph/osd/ceph-8/block) open stat got: (1) Operation not permitted 2024-09-10T19:42:35.946 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:35.943+0000 7fdc2452e640 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2024-09-10T19:42:38.314 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-09-10T19:42:38.344 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T19:42:38.434 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:38.432+0000 7f9f07afe640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2024-09-10T19:42:38.435 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:38.432+0000 7f9f07afe640 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2024-09-10T19:42:38.435 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:38.432+0000 7f9f07afe640 -1 bdev(0x55a41fcff800 /var/lib/ceph/osd/ceph-9/block) open stat got: (1) Operation not permitted 2024-09-10T19:42:38.435 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-10T19:42:38.432+0000 7f9f07afe640 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2024-09-10T19:42:40.563 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2024-09-10T19:42:40.591 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-10T19:42:40.592 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:40.592 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-10T19:42:40.659 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:40.659 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-10T19:42:40.726 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:40.726 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-10T19:42:40.792 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:40.793 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-10T19:42:40.859 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:40.859 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-10T19:42:40.925 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:40.925 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-10T19:42:40.990 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:40.990 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-10T19:42:41.055 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:41.055 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-10T19:42:41.120 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:41.120 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-10T19:42:41.185 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:41.185 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2024-09-10T19:42:41.249 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:41.250 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2024-09-10T19:42:41.314 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:41.314 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-10T19:42:41.371 INFO:tasks.ceph:Adding keys to all mons... 2024-09-10T19:42:41.371 DEBUG:teuthology.orchestra.run.smithi082:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-10T19:42:41.434 INFO:teuthology.orchestra.run.smithi082.stdout:[mgr.x] 2024-09-10T19:42:41.434 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQAVoeBmEagaIxAAmw5Wf0xip4YkjOozRAMoYQ== 2024-09-10T19:42:41.434 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.0] 2024-09-10T19:42:41.434 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQAZoeBmHi4bDRAA7USzi0ggdJRHdokwc/I1fQ== 2024-09-10T19:42:41.434 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.1] 2024-09-10T19:42:41.434 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQAboeBmLW6XGxAAKn/VIKtAUH1xvMDt3/yLsQ== 2024-09-10T19:42:41.434 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.2] 2024-09-10T19:42:41.435 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQAdoeBmuDApORAABSFr+P0IceN1uDTtSCUHug== 2024-09-10T19:42:41.435 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.3] 2024-09-10T19:42:41.435 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQAgoeBmRYo9DRAAmQdw+xf2BMISdCsJ+DdOog== 2024-09-10T19:42:41.435 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.4] 2024-09-10T19:42:41.435 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQAioeBmxMimGxAALqimqQbsxiViOdEDInppiw== 2024-09-10T19:42:41.435 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.5] 2024-09-10T19:42:41.435 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQAkoeBmV7WwOBAABsWZTDPjAn6ne/usQYKRrA== 2024-09-10T19:42:41.435 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.6] 2024-09-10T19:42:41.435 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQAnoeBmOs+mCxAAi5Qkp0WfuXsFMBZd0Q7p2A== 2024-09-10T19:42:41.435 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.7] 2024-09-10T19:42:41.435 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQApoeBme5zPGhAAoWlXSnqVrZvl7UxXZjjcyQ== 2024-09-10T19:42:41.435 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.8] 2024-09-10T19:42:41.435 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQAroeBm909KOBAA5Q92CE1pozJ32N+qjzSNrw== 2024-09-10T19:42:41.435 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.9] 2024-09-10T19:42:41.435 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQAuoeBm+oXVGRAAOgQZ85nmWaOnZKs2etrsyw== 2024-09-10T19:42:41.436 INFO:teuthology.orchestra.run.smithi082.stdout:[client.0] 2024-09-10T19:42:41.436 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQAVoeBmfpqyJRAATzuK/pJcKx6mviA1XBpXDQ== 2024-09-10T19:42:41.436 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-10T19:42:41.528 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T19:42:41.579 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T19:42:41.633 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T19:42:41.683 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T19:42:41.734 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T19:42:41.784 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T19:42:41.836 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T19:42:41.887 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T19:42:41.937 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T19:42:41.988 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T19:42:42.039 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-10T19:42:42.093 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-10T19:42:42.093 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-10T19:42:42.118 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-10T19:42:42.214 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-10T19:42:42.240 DEBUG:teuthology.orchestra.run.smithi082:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-10T19:42:42.295 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-10T19:42:42.295 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-10T19:42:42.295 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-10T19:42:42.337 INFO:tasks.ceph.mon.a:Started 2024-09-10T19:42:42.337 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph config set mgr mgr_pool false --force 2024-09-10T19:42:42.518 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-10T19:42:42.518 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-10T19:42:42.518 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-10T19:42:42.521 INFO:tasks.ceph.mgr.x:Started 2024-09-10T19:42:42.521 DEBUG:tasks.ceph:set 0 configs 2024-09-10T19:42:42.521 INFO:tasks.ceph:Setting crush tunables to default 2024-09-10T19:42:42.521 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd crush tunables default 2024-09-10T19:42:42.693 INFO:teuthology.orchestra.run.smithi082.stderr:adjusted tunables profile to default 2024-09-10T19:42:42.704 INFO:tasks.ceph:check_enable_crimson: False 2024-09-10T19:42:42.704 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-10T19:42:42.704 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:42.704 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-10T19:42:42.729 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:42.729 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-10T19:42:42.794 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:42.794 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-10T19:42:42.860 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:42.860 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-10T19:42:42.925 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:42.925 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-10T19:42:42.989 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:42.989 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-10T19:42:43.055 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:43.055 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-10T19:42:43.120 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:43.120 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-10T19:42:43.185 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:43.185 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2024-09-10T19:42:43.249 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:42:43.250 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2024-09-10T19:42:43.314 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new ad42803d-481b-49b9-9204-f2239ad7df2d 0 2024-09-10T19:42:43.532 INFO:teuthology.orchestra.run.smithi082.stdout:0 2024-09-10T19:42:43.543 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new c4fb1039-47d0-4f84-92ba-6d15d78a152d 1 2024-09-10T19:42:43.733 INFO:teuthology.orchestra.run.smithi082.stdout:1 2024-09-10T19:42:43.744 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new 79860c95-6c4e-4b14-bbbd-0655ed2aea42 2 2024-09-10T19:42:43.949 INFO:teuthology.orchestra.run.smithi082.stdout:2 2024-09-10T19:42:43.960 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new 7838eabe-f0fd-422d-91dc-2ffee0d0a458 3 2024-09-10T19:42:44.136 INFO:teuthology.orchestra.run.smithi082.stdout:3 2024-09-10T19:42:44.149 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new bac446ed-c130-445f-b889-9180350925e9 4 2024-09-10T19:42:44.325 INFO:teuthology.orchestra.run.smithi082.stdout:4 2024-09-10T19:42:44.337 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new c39e3a52-e1a0-4471-90fd-822b4f05199a 5 2024-09-10T19:42:44.541 INFO:teuthology.orchestra.run.smithi082.stdout:5 2024-09-10T19:42:44.552 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new 4bf59a66-0ff9-4245-8405-d9fe6216075f 6 2024-09-10T19:42:44.727 INFO:teuthology.orchestra.run.smithi082.stdout:6 2024-09-10T19:42:44.738 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new ad29323b-3d4d-4e3e-90cf-a2d57c95bce8 7 2024-09-10T19:42:44.914 INFO:teuthology.orchestra.run.smithi082.stdout:7 2024-09-10T19:42:44.925 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new 1f777d60-2f41-4f6b-a348-014322d1509e 8 2024-09-10T19:42:45.100 INFO:teuthology.orchestra.run.smithi082.stdout:8 2024-09-10T19:42:45.111 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new 5345f4dc-dd23-4e0a-8520-e9c675030b43 9 2024-09-10T19:42:45.288 INFO:teuthology.orchestra.run.smithi082.stdout:9 2024-09-10T19:42:45.300 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-10T19:42:45.300 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-10T19:42:45.302 INFO:tasks.ceph.osd.0:Started 2024-09-10T19:42:45.302 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-10T19:42:45.302 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-10T19:42:45.304 INFO:tasks.ceph.osd.1:Started 2024-09-10T19:42:45.304 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-10T19:42:45.304 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-10T19:42:45.306 INFO:tasks.ceph.osd.2:Started 2024-09-10T19:42:45.306 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-10T19:42:45.306 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-10T19:42:45.309 INFO:tasks.ceph.osd.3:Started 2024-09-10T19:42:45.309 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-10T19:42:45.309 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-10T19:42:45.311 INFO:tasks.ceph.osd.4:Started 2024-09-10T19:42:45.311 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-10T19:42:45.311 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-10T19:42:45.314 INFO:tasks.ceph.osd.5:Started 2024-09-10T19:42:45.314 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-10T19:42:45.314 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-10T19:42:45.318 INFO:tasks.ceph.osd.6:Started 2024-09-10T19:42:45.318 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-10T19:42:45.318 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-10T19:42:45.321 INFO:tasks.ceph.osd.7:Started 2024-09-10T19:42:45.321 INFO:tasks.ceph.osd.8:Restarting daemon 2024-09-10T19:42:45.321 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2024-09-10T19:42:45.325 INFO:tasks.ceph.osd.8:Started 2024-09-10T19:42:45.325 INFO:tasks.ceph.osd.9:Restarting daemon 2024-09-10T19:42:45.325 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2024-09-10T19:42:45.329 INFO:tasks.ceph.osd.9:Started 2024-09-10T19:42:45.329 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T19:42:45.599 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:45.600 INFO:teuthology.orchestra.run.smithi082.stdout:{"epoch":12,"fsid":"2964cb92-693e-4d63-a589-7427632fbccd","created":"2024-09-10T19:42:42.419999+0000","modified":"2024-09-10T19:42:45.287001+0000","last_up_change":"0.000000","last_in_change":"2024-09-10T19:42:45.287001+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ad42803d-481b-49b9-9204-f2239ad7df2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c4fb1039-47d0-4f84-92ba-6d15d78a152d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"79860c95-6c4e-4b14-bbbd-0655ed2aea42","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"7838eabe-f0fd-422d-91dc-2ffee0d0a458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bac446ed-c130-445f-b889-9180350925e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c39e3a52-e1a0-4471-90fd-822b4f05199a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4bf59a66-0ff9-4245-8405-d9fe6216075f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad29323b-3d4d-4e3e-90cf-a2d57c95bce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"1f777d60-2f41-4f6b-a348-014322d1509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"5345f4dc-dd23-4e0a-8520-e9c675030b43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T19:42:45.611 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-10T19:42:45.611 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-10T19:42:46.596 INFO:tasks.ceph.osd.0.smithi082.stderr:2024-09-10T19:42:46.594+0000 7fdece02f640 -1 Falling back to public interface 2024-09-10T19:42:46.615 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-10T19:42:46.651 INFO:tasks.ceph.osd.8.smithi082.stderr:2024-09-10T19:42:46.649+0000 7f6159824640 -1 Falling back to public interface 2024-09-10T19:42:46.774 INFO:teuthology.misc.health.smithi082.stdout: 2024-09-10T19:42:46.774 INFO:teuthology.misc.health.smithi082.stdout:{"epoch":12,"fsid":"2964cb92-693e-4d63-a589-7427632fbccd","created":"2024-09-10T19:42:42.419999+0000","modified":"2024-09-10T19:42:45.287001+0000","last_up_change":"0.000000","last_in_change":"2024-09-10T19:42:45.287001+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ad42803d-481b-49b9-9204-f2239ad7df2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c4fb1039-47d0-4f84-92ba-6d15d78a152d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"79860c95-6c4e-4b14-bbbd-0655ed2aea42","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"7838eabe-f0fd-422d-91dc-2ffee0d0a458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bac446ed-c130-445f-b889-9180350925e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c39e3a52-e1a0-4471-90fd-822b4f05199a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4bf59a66-0ff9-4245-8405-d9fe6216075f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad29323b-3d4d-4e3e-90cf-a2d57c95bce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"1f777d60-2f41-4f6b-a348-014322d1509e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"5345f4dc-dd23-4e0a-8520-e9c675030b43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T19:42:46.784 DEBUG:teuthology.misc:0 of 10 OSDs are up 2024-09-10T19:42:46.879 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-10T19:42:46.877+0000 7f0623ab0640 -1 Falling back to public interface 2024-09-10T19:42:46.880 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-10T19:42:46.878+0000 7fc453618640 -1 Falling back to public interface 2024-09-10T19:42:46.882 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-10T19:42:46.880+0000 7f9ca7c0f640 -1 Falling back to public interface 2024-09-10T19:42:47.073 INFO:tasks.ceph.osd.1.smithi082.stderr:2024-09-10T19:42:47.070+0000 7fceab61c640 -1 Falling back to public interface 2024-09-10T19:42:47.105 INFO:tasks.ceph.osd.2.smithi082.stderr:2024-09-10T19:42:47.103+0000 7fae91a11640 -1 Falling back to public interface 2024-09-10T19:42:47.157 INFO:tasks.ceph.osd.9.smithi082.stderr:2024-09-10T19:42:47.155+0000 7fa16029c640 -1 Falling back to public interface 2024-09-10T19:42:47.336 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-09-10T19:42:47.334+0000 7f6d47c16640 -1 Falling back to public interface 2024-09-10T19:42:47.373 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-09-10T19:42:47.371+0000 7f247ae2f640 -1 Falling back to public interface 2024-09-10T19:42:48.009 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-09-10T19:42:48.007+0000 7f37b40ae640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-10T19:42:49.666 INFO:tasks.ceph.osd.8.smithi082.stderr:2024-09-10T19:42:49.664+0000 7f6159824640 -1 osd.8 0 log_to_monitors true 2024-09-10T19:42:49.899 INFO:tasks.ceph.osd.0.smithi082.stderr:2024-09-10T19:42:49.896+0000 7fdece02f640 -1 osd.0 0 log_to_monitors true 2024-09-10T19:42:49.915 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-10T19:42:49.912+0000 7fc453618640 -1 osd.5 0 log_to_monitors true 2024-09-10T19:42:50.080 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-09-10T19:42:50.078+0000 7f6d47c16640 -1 osd.6 0 log_to_monitors true 2024-09-10T19:42:50.159 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-10T19:42:50.157+0000 7f9ca7c0f640 -1 osd.4 0 log_to_monitors true 2024-09-10T19:42:50.175 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-10T19:42:50.172+0000 7f0623ab0640 -1 osd.3 0 log_to_monitors true 2024-09-10T19:42:50.372 INFO:tasks.ceph.osd.1.smithi082.stderr:2024-09-10T19:42:50.370+0000 7fceab61c640 -1 osd.1 0 log_to_monitors true 2024-09-10T19:42:50.446 INFO:tasks.ceph.osd.9.smithi082.stderr:2024-09-10T19:42:50.444+0000 7fa16029c640 -1 osd.9 0 log_to_monitors true 2024-09-10T19:42:50.620 INFO:tasks.ceph.osd.2.smithi082.stderr:2024-09-10T19:42:50.618+0000 7fae91a11640 -1 osd.2 0 log_to_monitors true 2024-09-10T19:42:50.665 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-09-10T19:42:50.663+0000 7f247ae2f640 -1 osd.7 0 log_to_monitors true 2024-09-10T19:42:51.036 INFO:tasks.ceph.osd.0.smithi082.stderr:2024-09-10T19:42:51.034+0000 7fdec9fbe640 -1 osd.0 0 waiting for initial osdmap 2024-09-10T19:42:51.130 INFO:tasks.ceph.osd.0.smithi082.stderr:2024-09-10T19:42:51.128+0000 7fdec5de5640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:42:51.917 INFO:tasks.ceph.osd.8.smithi082.stderr:2024-09-10T19:42:51.915+0000 7f61557b3640 -1 osd.8 0 waiting for initial osdmap 2024-09-10T19:42:51.938 INFO:tasks.ceph.osd.8.smithi082.stderr:2024-09-10T19:42:51.936+0000 7f614d5d2640 -1 osd.8 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:42:52.197 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-09-10T19:42:52.195+0000 7f6d43624640 -1 osd.6 0 waiting for initial osdmap 2024-09-10T19:42:52.203 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-10T19:42:52.201+0000 7f061fa3f640 -1 osd.3 0 waiting for initial osdmap 2024-09-10T19:42:52.206 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-10T19:42:52.204+0000 7f061b866640 -1 osd.3 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:42:52.206 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-10T19:42:52.205+0000 7f9ca351e640 -1 osd.4 0 waiting for initial osdmap 2024-09-10T19:42:52.210 INFO:tasks.ceph.osd.2.smithi082.stderr:2024-09-10T19:42:52.208+0000 7fae8d392640 -1 osd.2 0 waiting for initial osdmap 2024-09-10T19:42:52.353 INFO:tasks.ceph.osd.2.smithi082.stderr:2024-09-10T19:42:52.351+0000 7fae891b9640 -1 osd.2 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:42:52.380 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-09-10T19:42:52.378+0000 7f6d3f44b640 -1 osd.6 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:42:52.412 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-10T19:42:52.409+0000 7f9c9f345640 -1 osd.4 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:42:52.867 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-10T19:42:52.864+0000 7fc44ef7a640 -1 osd.5 0 waiting for initial osdmap 2024-09-10T19:42:53.039 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-10T19:42:53.037+0000 7fc446d99640 -1 osd.5 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:42:53.224 INFO:tasks.ceph.osd.1.smithi082.stderr:2024-09-10T19:42:53.222+0000 7fcea6f0d640 -1 osd.1 0 waiting for initial osdmap 2024-09-10T19:42:53.234 INFO:tasks.ceph.osd.1.smithi082.stderr:2024-09-10T19:42:53.232+0000 7fce9ed2c640 -1 osd.1 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:42:53.316 INFO:tasks.ceph.osd.9.smithi082.stderr:2024-09-10T19:42:53.314+0000 7fa15c22b640 -1 osd.9 0 waiting for initial osdmap 2024-09-10T19:42:53.316 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-09-10T19:42:53.314+0000 7f2476385640 -1 osd.7 0 waiting for initial osdmap 2024-09-10T19:42:53.324 INFO:tasks.ceph.osd.9.smithi082.stderr:2024-09-10T19:42:53.322+0000 7fa15404a640 -1 osd.9 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:42:53.328 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-09-10T19:42:53.326+0000 7f246e1a4640 -1 osd.7 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:42:53.790 DEBUG:teuthology.orchestra.run.smithi082:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-10T19:42:54.078 INFO:teuthology.misc.health.smithi082.stdout: 2024-09-10T19:42:54.078 INFO:teuthology.misc.health.smithi082.stdout:{"epoch":17,"fsid":"2964cb92-693e-4d63-a589-7427632fbccd","created":"2024-09-10T19:42:42.419999+0000","modified":"2024-09-10T19:42:54.043043+0000","last_up_change":"2024-09-10T19:42:54.043043+0000","last_in_change":"2024-09-10T19:42:45.287001+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ad42803d-481b-49b9-9204-f2239ad7df2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1663731654},{"type":"v1","addr":"172.21.15.82:6801","nonce":1663731654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1663731654},{"type":"v1","addr":"172.21.15.82:6803","nonce":1663731654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1663731654},{"type":"v1","addr":"172.21.15.82:6807","nonce":1663731654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1663731654},{"type":"v1","addr":"172.21.15.82:6805","nonce":1663731654}]},"public_addr":"172.21.15.82:6801/1663731654","cluster_addr":"172.21.15.82:6803/1663731654","heartbeat_back_addr":"172.21.15.82:6807/1663731654","heartbeat_front_addr":"172.21.15.82:6805/1663731654","state":["exists","up"]},{"osd":1,"uuid":"c4fb1039-47d0-4f84-92ba-6d15d78a152d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6842","nonce":3441180664},{"type":"v1","addr":"172.21.15.82:6843","nonce":3441180664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6844","nonce":3441180664},{"type":"v1","addr":"172.21.15.82:6845","nonce":3441180664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6848","nonce":3441180664},{"type":"v1","addr":"172.21.15.82:6849","nonce":3441180664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6846","nonce":3441180664},{"type":"v1","addr":"172.21.15.82:6847","nonce":3441180664}]},"public_addr":"172.21.15.82:6843/3441180664","cluster_addr":"172.21.15.82:6845/3441180664","heartbeat_back_addr":"172.21.15.82:6849/3441180664","heartbeat_front_addr":"172.21.15.82:6847/3441180664","state":["exists","up"]},{"osd":2,"uuid":"79860c95-6c4e-4b14-bbbd-0655ed2aea42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6850","nonce":3980847884},{"type":"v1","addr":"172.21.15.82:6851","nonce":3980847884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6852","nonce":3980847884},{"type":"v1","addr":"172.21.15.82:6853","nonce":3980847884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6856","nonce":3980847884},{"type":"v1","addr":"172.21.15.82:6857","nonce":3980847884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6854","nonce":3980847884},{"type":"v1","addr":"172.21.15.82:6855","nonce":3980847884}]},"public_addr":"172.21.15.82:6851/3980847884","cluster_addr":"172.21.15.82:6853/3980847884","heartbeat_back_addr":"172.21.15.82:6857/3980847884","heartbeat_front_addr":"172.21.15.82:6855/3980847884","state":["exists","up"]},{"osd":3,"uuid":"7838eabe-f0fd-422d-91dc-2ffee0d0a458","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":3682993015},{"type":"v1","addr":"172.21.15.82:6817","nonce":3682993015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":3682993015},{"type":"v1","addr":"172.21.15.82:6819","nonce":3682993015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3682993015},{"type":"v1","addr":"172.21.15.82:6826","nonce":3682993015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":3682993015},{"type":"v1","addr":"172.21.15.82:6822","nonce":3682993015}]},"public_addr":"172.21.15.82:6817/3682993015","cluster_addr":"172.21.15.82:6819/3682993015","heartbeat_back_addr":"172.21.15.82:6826/3682993015","heartbeat_front_addr":"172.21.15.82:6822/3682993015","state":["exists","up"]},{"osd":4,"uuid":"bac446ed-c130-445f-b889-9180350925e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":107394753},{"type":"v1","addr":"172.21.15.82:6832","nonce":107394753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":107394753},{"type":"v1","addr":"172.21.15.82:6835","nonce":107394753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":107394753},{"type":"v1","addr":"172.21.15.82:6839","nonce":107394753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":107394753},{"type":"v1","addr":"172.21.15.82:6837","nonce":107394753}]},"public_addr":"172.21.15.82:6832/107394753","cluster_addr":"172.21.15.82:6835/107394753","heartbeat_back_addr":"172.21.15.82:6839/107394753","heartbeat_front_addr":"172.21.15.82:6837/107394753","state":["exists","up"]},{"osd":5,"uuid":"c39e3a52-e1a0-4471-90fd-822b4f05199a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6821","nonce":3810102128},{"type":"v1","addr":"172.21.15.82:6823","nonce":3810102128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6825","nonce":3810102128},{"type":"v1","addr":"172.21.15.82:6827","nonce":3810102128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6831","nonce":3810102128},{"type":"v1","addr":"172.21.15.82:6833","nonce":3810102128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3810102128},{"type":"v1","addr":"172.21.15.82:6829","nonce":3810102128}]},"public_addr":"172.21.15.82:6823/3810102128","cluster_addr":"172.21.15.82:6827/3810102128","heartbeat_back_addr":"172.21.15.82:6833/3810102128","heartbeat_front_addr":"172.21.15.82:6829/3810102128","state":["exists","up"]},{"osd":6,"uuid":"4bf59a66-0ff9-4245-8405-d9fe6216075f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6866","nonce":454186391},{"type":"v1","addr":"172.21.15.82:6867","nonce":454186391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6868","nonce":454186391},{"type":"v1","addr":"172.21.15.82:6869","nonce":454186391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6872","nonce":454186391},{"type":"v1","addr":"172.21.15.82:6873","nonce":454186391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6870","nonce":454186391},{"type":"v1","addr":"172.21.15.82:6871","nonce":454186391}]},"public_addr":"172.21.15.82:6867/454186391","cluster_addr":"172.21.15.82:6869/454186391","heartbeat_back_addr":"172.21.15.82:6873/454186391","heartbeat_front_addr":"172.21.15.82:6871/454186391","state":["exists","up"]},{"osd":7,"uuid":"ad29323b-3d4d-4e3e-90cf-a2d57c95bce8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6874","nonce":192219834},{"type":"v1","addr":"172.21.15.82:6875","nonce":192219834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6876","nonce":192219834},{"type":"v1","addr":"172.21.15.82:6877","nonce":192219834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6880","nonce":192219834},{"type":"v1","addr":"172.21.15.82:6881","nonce":192219834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6878","nonce":192219834},{"type":"v1","addr":"172.21.15.82:6879","nonce":192219834}]},"public_addr":"172.21.15.82:6875/192219834","cluster_addr":"172.21.15.82:6877/192219834","heartbeat_back_addr":"172.21.15.82:6881/192219834","heartbeat_front_addr":"172.21.15.82:6879/192219834","state":["exists","up"]},{"osd":8,"uuid":"1f777d60-2f41-4f6b-a348-014322d1509e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":691756156},{"type":"v1","addr":"172.21.15.82:6809","nonce":691756156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":691756156},{"type":"v1","addr":"172.21.15.82:6811","nonce":691756156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":691756156},{"type":"v1","addr":"172.21.15.82:6815","nonce":691756156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":691756156},{"type":"v1","addr":"172.21.15.82:6813","nonce":691756156}]},"public_addr":"172.21.15.82:6809/691756156","cluster_addr":"172.21.15.82:6811/691756156","heartbeat_back_addr":"172.21.15.82:6815/691756156","heartbeat_front_addr":"172.21.15.82:6813/691756156","state":["exists","up"]},{"osd":9,"uuid":"5345f4dc-dd23-4e0a-8520-e9c675030b43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6858","nonce":2504847011},{"type":"v1","addr":"172.21.15.82:6859","nonce":2504847011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6860","nonce":2504847011},{"type":"v1","addr":"172.21.15.82:6861","nonce":2504847011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6864","nonce":2504847011},{"type":"v1","addr":"172.21.15.82:6865","nonce":2504847011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6862","nonce":2504847011},{"type":"v1","addr":"172.21.15.82:6863","nonce":2504847011}]},"public_addr":"172.21.15.82:6859/2504847011","cluster_addr":"172.21.15.82:6861/2504847011","heartbeat_back_addr":"172.21.15.82:6865/2504847011","heartbeat_front_addr":"172.21.15.82:6863/2504847011","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:50.873734+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:51.637626+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:51.138513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:51.176514+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:51.068017+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:50.655627+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T19:42:54.089 DEBUG:teuthology.misc:10 of 10 OSDs are up 2024-09-10T19:42:54.089 INFO:tasks.ceph:Creating RBD pool 2024-09-10T19:42:54.089 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-10T19:42:55.231 INFO:teuthology.orchestra.run.smithi082.stderr:pool 'rbd' created 2024-09-10T19:42:55.245 DEBUG:teuthology.orchestra.run.smithi082:> rbd --cluster ceph pool init rbd 2024-09-10T19:42:58.252 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-10T19:42:58.252 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-10T19:42:58.253 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-10T19:42:58.547 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:58.559 INFO:teuthology.orchestra.run.smithi082.stdout:[{"version":7,"timestamp":"2024-09-10T19:42:53.317982+0000","name":"","changes":[{"name":"osd.7/osd_mclock_max_capacity_iops_ssd","new_value":"17816.700538"}]}] 2024-09-10T19:42:58.560 INFO:tasks.ceph_manager:config epoch is 7 2024-09-10T19:42:58.560 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-10T19:42:58.560 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-10T19:42:58.560 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-10T19:42:58.891 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:58.902 INFO:teuthology.orchestra.run.smithi082.stdout:{"epoch":4,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6840","nonce":932225513},{"type":"v1","addr":"172.21.15.82:6841","nonce":932225513}]},"active_addr":"172.21.15.82:6841/932225513","active_change":"2024-09-10T19:42:47.002072+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.82:0","nonce":1033370753}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.82:0","nonce":2015136187}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.82:0","nonce":875941709}]}]} 2024-09-10T19:42:58.905 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-10T19:42:58.905 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-10T19:42:58.905 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T19:42:59.187 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:59.187 INFO:teuthology.orchestra.run.smithi082.stdout:{"epoch":21,"fsid":"2964cb92-693e-4d63-a589-7427632fbccd","created":"2024-09-10T19:42:42.419999+0000","modified":"2024-09-10T19:42:58.237487+0000","last_up_change":"2024-09-10T19:42:54.043043+0000","last_in_change":"2024-09-10T19:42:45.287001+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-10T19:42:54.375436+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":21,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":0.89999997615814209,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1.0000001192092896,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad42803d-481b-49b9-9204-f2239ad7df2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1663731654},{"type":"v1","addr":"172.21.15.82:6801","nonce":1663731654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1663731654},{"type":"v1","addr":"172.21.15.82:6803","nonce":1663731654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1663731654},{"type":"v1","addr":"172.21.15.82:6807","nonce":1663731654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1663731654},{"type":"v1","addr":"172.21.15.82:6805","nonce":1663731654}]},"public_addr":"172.21.15.82:6801/1663731654","cluster_addr":"172.21.15.82:6803/1663731654","heartbeat_back_addr":"172.21.15.82:6807/1663731654","heartbeat_front_addr":"172.21.15.82:6805/1663731654","state":["exists","up"]},{"osd":1,"uuid":"c4fb1039-47d0-4f84-92ba-6d15d78a152d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6842","nonce":3441180664},{"type":"v1","addr":"172.21.15.82:6843","nonce":3441180664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6844","nonce":3441180664},{"type":"v1","addr":"172.21.15.82:6845","nonce":3441180664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6848","nonce":3441180664},{"type":"v1","addr":"172.21.15.82:6849","nonce":3441180664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6846","nonce":3441180664},{"type":"v1","addr":"172.21.15.82:6847","nonce":3441180664}]},"public_addr":"172.21.15.82:6843/3441180664","cluster_addr":"172.21.15.82:6845/3441180664","heartbeat_back_addr":"172.21.15.82:6849/3441180664","heartbeat_front_addr":"172.21.15.82:6847/3441180664","state":["exists","up"]},{"osd":2,"uuid":"79860c95-6c4e-4b14-bbbd-0655ed2aea42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6850","nonce":3980847884},{"type":"v1","addr":"172.21.15.82:6851","nonce":3980847884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6852","nonce":3980847884},{"type":"v1","addr":"172.21.15.82:6853","nonce":3980847884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6856","nonce":3980847884},{"type":"v1","addr":"172.21.15.82:6857","nonce":3980847884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6854","nonce":3980847884},{"type":"v1","addr":"172.21.15.82:6855","nonce":3980847884}]},"public_addr":"172.21.15.82:6851/3980847884","cluster_addr":"172.21.15.82:6853/3980847884","heartbeat_back_addr":"172.21.15.82:6857/3980847884","heartbeat_front_addr":"172.21.15.82:6855/3980847884","state":["exists","up"]},{"osd":3,"uuid":"7838eabe-f0fd-422d-91dc-2ffee0d0a458","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":3682993015},{"type":"v1","addr":"172.21.15.82:6817","nonce":3682993015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":3682993015},{"type":"v1","addr":"172.21.15.82:6819","nonce":3682993015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3682993015},{"type":"v1","addr":"172.21.15.82:6826","nonce":3682993015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":3682993015},{"type":"v1","addr":"172.21.15.82:6822","nonce":3682993015}]},"public_addr":"172.21.15.82:6817/3682993015","cluster_addr":"172.21.15.82:6819/3682993015","heartbeat_back_addr":"172.21.15.82:6826/3682993015","heartbeat_front_addr":"172.21.15.82:6822/3682993015","state":["exists","up"]},{"osd":4,"uuid":"bac446ed-c130-445f-b889-9180350925e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":107394753},{"type":"v1","addr":"172.21.15.82:6832","nonce":107394753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":107394753},{"type":"v1","addr":"172.21.15.82:6835","nonce":107394753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":107394753},{"type":"v1","addr":"172.21.15.82:6839","nonce":107394753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":107394753},{"type":"v1","addr":"172.21.15.82:6837","nonce":107394753}]},"public_addr":"172.21.15.82:6832/107394753","cluster_addr":"172.21.15.82:6835/107394753","heartbeat_back_addr":"172.21.15.82:6839/107394753","heartbeat_front_addr":"172.21.15.82:6837/107394753","state":["exists","up"]},{"osd":5,"uuid":"c39e3a52-e1a0-4471-90fd-822b4f05199a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6821","nonce":3810102128},{"type":"v1","addr":"172.21.15.82:6823","nonce":3810102128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6825","nonce":3810102128},{"type":"v1","addr":"172.21.15.82:6827","nonce":3810102128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6831","nonce":3810102128},{"type":"v1","addr":"172.21.15.82:6833","nonce":3810102128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3810102128},{"type":"v1","addr":"172.21.15.82:6829","nonce":3810102128}]},"public_addr":"172.21.15.82:6823/3810102128","cluster_addr":"172.21.15.82:6827/3810102128","heartbeat_back_addr":"172.21.15.82:6833/3810102128","heartbeat_front_addr":"172.21.15.82:6829/3810102128","state":["exists","up"]},{"osd":6,"uuid":"4bf59a66-0ff9-4245-8405-d9fe6216075f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6866","nonce":454186391},{"type":"v1","addr":"172.21.15.82:6867","nonce":454186391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6868","nonce":454186391},{"type":"v1","addr":"172.21.15.82:6869","nonce":454186391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6872","nonce":454186391},{"type":"v1","addr":"172.21.15.82:6873","nonce":454186391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6870","nonce":454186391},{"type":"v1","addr":"172.21.15.82:6871","nonce":454186391}]},"public_addr":"172.21.15.82:6867/454186391","cluster_addr":"172.21.15.82:6869/454186391","heartbeat_back_addr":"172.21.15.82:6873/454186391","heartbeat_front_addr":"172.21.15.82:6871/454186391","state":["exists","up"]},{"osd":7,"uuid":"ad29323b-3d4d-4e3e-90cf-a2d57c95bce8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6874","nonce":192219834},{"type":"v1","addr":"172.21.15.82:6875","nonce":192219834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6876","nonce":192219834},{"type":"v1","addr":"172.21.15.82:6877","nonce":192219834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6880","nonce":192219834},{"type":"v1","addr":"172.21.15.82:6881","nonce":192219834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6878","nonce":192219834},{"type":"v1","addr":"172.21.15.82:6879","nonce":192219834}]},"public_addr":"172.21.15.82:6875/192219834","cluster_addr":"172.21.15.82:6877/192219834","heartbeat_back_addr":"172.21.15.82:6881/192219834","heartbeat_front_addr":"172.21.15.82:6879/192219834","state":["exists","up"]},{"osd":8,"uuid":"1f777d60-2f41-4f6b-a348-014322d1509e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":691756156},{"type":"v1","addr":"172.21.15.82:6809","nonce":691756156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":691756156},{"type":"v1","addr":"172.21.15.82:6811","nonce":691756156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":691756156},{"type":"v1","addr":"172.21.15.82:6815","nonce":691756156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":691756156},{"type":"v1","addr":"172.21.15.82:6813","nonce":691756156}]},"public_addr":"172.21.15.82:6809/691756156","cluster_addr":"172.21.15.82:6811/691756156","heartbeat_back_addr":"172.21.15.82:6815/691756156","heartbeat_front_addr":"172.21.15.82:6813/691756156","state":["exists","up"]},{"osd":9,"uuid":"5345f4dc-dd23-4e0a-8520-e9c675030b43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6858","nonce":2504847011},{"type":"v1","addr":"172.21.15.82:6859","nonce":2504847011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6860","nonce":2504847011},{"type":"v1","addr":"172.21.15.82:6861","nonce":2504847011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6864","nonce":2504847011},{"type":"v1","addr":"172.21.15.82:6865","nonce":2504847011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6862","nonce":2504847011},{"type":"v1","addr":"172.21.15.82:6863","nonce":2504847011}]},"public_addr":"172.21.15.82:6859/2504847011","cluster_addr":"172.21.15.82:6861/2504847011","heartbeat_back_addr":"172.21.15.82:6865/2504847011","heartbeat_front_addr":"172.21.15.82:6863/2504847011","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:50.873734+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:51.362765+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:51.637626+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:51.138513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:51.176514+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:50.914217+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:51.068017+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:51.625988+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:50.655627+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:51.405482+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T19:42:59.200 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-10T19:42:59.200 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T19:42:59.484 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:42:59.484 INFO:teuthology.orchestra.run.smithi082.stdout:{"epoch":21,"fsid":"2964cb92-693e-4d63-a589-7427632fbccd","created":"2024-09-10T19:42:42.419999+0000","modified":"2024-09-10T19:42:58.237487+0000","last_up_change":"2024-09-10T19:42:54.043043+0000","last_in_change":"2024-09-10T19:42:45.287001+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-10T19:42:54.375436+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":21,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":0.89999997615814209,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1.0000001192092896,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad42803d-481b-49b9-9204-f2239ad7df2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1663731654},{"type":"v1","addr":"172.21.15.82:6801","nonce":1663731654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1663731654},{"type":"v1","addr":"172.21.15.82:6803","nonce":1663731654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1663731654},{"type":"v1","addr":"172.21.15.82:6807","nonce":1663731654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1663731654},{"type":"v1","addr":"172.21.15.82:6805","nonce":1663731654}]},"public_addr":"172.21.15.82:6801/1663731654","cluster_addr":"172.21.15.82:6803/1663731654","heartbeat_back_addr":"172.21.15.82:6807/1663731654","heartbeat_front_addr":"172.21.15.82:6805/1663731654","state":["exists","up"]},{"osd":1,"uuid":"c4fb1039-47d0-4f84-92ba-6d15d78a152d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6842","nonce":3441180664},{"type":"v1","addr":"172.21.15.82:6843","nonce":3441180664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6844","nonce":3441180664},{"type":"v1","addr":"172.21.15.82:6845","nonce":3441180664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6848","nonce":3441180664},{"type":"v1","addr":"172.21.15.82:6849","nonce":3441180664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6846","nonce":3441180664},{"type":"v1","addr":"172.21.15.82:6847","nonce":3441180664}]},"public_addr":"172.21.15.82:6843/3441180664","cluster_addr":"172.21.15.82:6845/3441180664","heartbeat_back_addr":"172.21.15.82:6849/3441180664","heartbeat_front_addr":"172.21.15.82:6847/3441180664","state":["exists","up"]},{"osd":2,"uuid":"79860c95-6c4e-4b14-bbbd-0655ed2aea42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6850","nonce":3980847884},{"type":"v1","addr":"172.21.15.82:6851","nonce":3980847884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6852","nonce":3980847884},{"type":"v1","addr":"172.21.15.82:6853","nonce":3980847884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6856","nonce":3980847884},{"type":"v1","addr":"172.21.15.82:6857","nonce":3980847884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6854","nonce":3980847884},{"type":"v1","addr":"172.21.15.82:6855","nonce":3980847884}]},"public_addr":"172.21.15.82:6851/3980847884","cluster_addr":"172.21.15.82:6853/3980847884","heartbeat_back_addr":"172.21.15.82:6857/3980847884","heartbeat_front_addr":"172.21.15.82:6855/3980847884","state":["exists","up"]},{"osd":3,"uuid":"7838eabe-f0fd-422d-91dc-2ffee0d0a458","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":3682993015},{"type":"v1","addr":"172.21.15.82:6817","nonce":3682993015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":3682993015},{"type":"v1","addr":"172.21.15.82:6819","nonce":3682993015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3682993015},{"type":"v1","addr":"172.21.15.82:6826","nonce":3682993015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":3682993015},{"type":"v1","addr":"172.21.15.82:6822","nonce":3682993015}]},"public_addr":"172.21.15.82:6817/3682993015","cluster_addr":"172.21.15.82:6819/3682993015","heartbeat_back_addr":"172.21.15.82:6826/3682993015","heartbeat_front_addr":"172.21.15.82:6822/3682993015","state":["exists","up"]},{"osd":4,"uuid":"bac446ed-c130-445f-b889-9180350925e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":107394753},{"type":"v1","addr":"172.21.15.82:6832","nonce":107394753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":107394753},{"type":"v1","addr":"172.21.15.82:6835","nonce":107394753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":107394753},{"type":"v1","addr":"172.21.15.82:6839","nonce":107394753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":107394753},{"type":"v1","addr":"172.21.15.82:6837","nonce":107394753}]},"public_addr":"172.21.15.82:6832/107394753","cluster_addr":"172.21.15.82:6835/107394753","heartbeat_back_addr":"172.21.15.82:6839/107394753","heartbeat_front_addr":"172.21.15.82:6837/107394753","state":["exists","up"]},{"osd":5,"uuid":"c39e3a52-e1a0-4471-90fd-822b4f05199a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6821","nonce":3810102128},{"type":"v1","addr":"172.21.15.82:6823","nonce":3810102128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6825","nonce":3810102128},{"type":"v1","addr":"172.21.15.82:6827","nonce":3810102128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6831","nonce":3810102128},{"type":"v1","addr":"172.21.15.82:6833","nonce":3810102128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3810102128},{"type":"v1","addr":"172.21.15.82:6829","nonce":3810102128}]},"public_addr":"172.21.15.82:6823/3810102128","cluster_addr":"172.21.15.82:6827/3810102128","heartbeat_back_addr":"172.21.15.82:6833/3810102128","heartbeat_front_addr":"172.21.15.82:6829/3810102128","state":["exists","up"]},{"osd":6,"uuid":"4bf59a66-0ff9-4245-8405-d9fe6216075f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6866","nonce":454186391},{"type":"v1","addr":"172.21.15.82:6867","nonce":454186391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6868","nonce":454186391},{"type":"v1","addr":"172.21.15.82:6869","nonce":454186391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6872","nonce":454186391},{"type":"v1","addr":"172.21.15.82:6873","nonce":454186391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6870","nonce":454186391},{"type":"v1","addr":"172.21.15.82:6871","nonce":454186391}]},"public_addr":"172.21.15.82:6867/454186391","cluster_addr":"172.21.15.82:6869/454186391","heartbeat_back_addr":"172.21.15.82:6873/454186391","heartbeat_front_addr":"172.21.15.82:6871/454186391","state":["exists","up"]},{"osd":7,"uuid":"ad29323b-3d4d-4e3e-90cf-a2d57c95bce8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6874","nonce":192219834},{"type":"v1","addr":"172.21.15.82:6875","nonce":192219834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6876","nonce":192219834},{"type":"v1","addr":"172.21.15.82:6877","nonce":192219834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6880","nonce":192219834},{"type":"v1","addr":"172.21.15.82:6881","nonce":192219834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6878","nonce":192219834},{"type":"v1","addr":"172.21.15.82:6879","nonce":192219834}]},"public_addr":"172.21.15.82:6875/192219834","cluster_addr":"172.21.15.82:6877/192219834","heartbeat_back_addr":"172.21.15.82:6881/192219834","heartbeat_front_addr":"172.21.15.82:6879/192219834","state":["exists","up"]},{"osd":8,"uuid":"1f777d60-2f41-4f6b-a348-014322d1509e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":691756156},{"type":"v1","addr":"172.21.15.82:6809","nonce":691756156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":691756156},{"type":"v1","addr":"172.21.15.82:6811","nonce":691756156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":691756156},{"type":"v1","addr":"172.21.15.82:6815","nonce":691756156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":691756156},{"type":"v1","addr":"172.21.15.82:6813","nonce":691756156}]},"public_addr":"172.21.15.82:6809/691756156","cluster_addr":"172.21.15.82:6811/691756156","heartbeat_back_addr":"172.21.15.82:6815/691756156","heartbeat_front_addr":"172.21.15.82:6813/691756156","state":["exists","up"]},{"osd":9,"uuid":"5345f4dc-dd23-4e0a-8520-e9c675030b43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6858","nonce":2504847011},{"type":"v1","addr":"172.21.15.82:6859","nonce":2504847011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6860","nonce":2504847011},{"type":"v1","addr":"172.21.15.82:6861","nonce":2504847011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6864","nonce":2504847011},{"type":"v1","addr":"172.21.15.82:6865","nonce":2504847011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6862","nonce":2504847011},{"type":"v1","addr":"172.21.15.82:6863","nonce":2504847011}]},"public_addr":"172.21.15.82:6859/2504847011","cluster_addr":"172.21.15.82:6861/2504847011","heartbeat_back_addr":"172.21.15.82:6865/2504847011","heartbeat_front_addr":"172.21.15.82:6863/2504847011","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:50.873734+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:51.362765+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:51.637626+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:51.138513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:51.176514+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:50.914217+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:51.068017+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:51.625988+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:50.655627+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:51.405482+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T19:42:59.497 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-10T19:42:59.497 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-10T19:42:59.498 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-10T19:42:59.498 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-10T19:42:59.498 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-10T19:42:59.499 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-10T19:42:59.499 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-10T19:42:59.500 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-10T19:42:59.500 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-10T19:42:59.500 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-10T19:42:59.765 INFO:teuthology.orchestra.run.smithi082.stdout:64424509445 2024-09-10T19:42:59.765 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-10T19:42:59.771 INFO:teuthology.orchestra.run.smithi082.stdout:73014444037 2024-09-10T19:42:59.771 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-10T19:42:59.775 INFO:teuthology.orchestra.run.smithi082.stdout:68719476739 2024-09-10T19:42:59.776 INFO:teuthology.orchestra.run.smithi082.stdout:68719476740 2024-09-10T19:42:59.776 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-10T19:42:59.777 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-10T19:42:59.786 INFO:teuthology.orchestra.run.smithi082.stdout:73014444035 2024-09-10T19:42:59.786 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-10T19:42:59.787 INFO:teuthology.orchestra.run.smithi082.stdout:68719476739 2024-09-10T19:42:59.788 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-10T19:42:59.792 INFO:teuthology.orchestra.run.smithi082.stdout:64424509444 2024-09-10T19:42:59.793 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-10T19:42:59.804 INFO:teuthology.orchestra.run.smithi082.stdout:68719476739 2024-09-10T19:42:59.805 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-10T19:42:59.818 INFO:teuthology.orchestra.run.smithi082.stdout:73014444036 2024-09-10T19:42:59.818 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-10T19:42:59.832 INFO:teuthology.orchestra.run.smithi082.stdout:73014444035 2024-09-10T19:42:59.833 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-10T19:43:00.451 INFO:teuthology.orchestra.run.smithi082.stdout:73014444036 2024-09-10T19:43:00.463 INFO:teuthology.orchestra.run.smithi082.stdout:68719476738 2024-09-10T19:43:00.482 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444037 got 73014444036 for osd.5 2024-09-10T19:43:00.486 INFO:teuthology.orchestra.run.smithi082.stdout:73014444033 2024-09-10T19:43:00.487 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476739 got 68719476738 for osd.3 2024-09-10T19:43:00.491 INFO:teuthology.orchestra.run.smithi082.stdout:64424509444 2024-09-10T19:43:00.507 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444035 got 73014444033 for osd.1 2024-09-10T19:43:00.508 INFO:teuthology.orchestra.run.smithi082.stdout:68719476739 2024-09-10T19:43:00.509 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509444 for osd.8 2024-09-10T19:43:00.511 INFO:teuthology.orchestra.run.smithi082.stdout:68719476738 2024-09-10T19:43:00.528 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476740 got 68719476739 for osd.4 2024-09-10T19:43:00.528 INFO:teuthology.orchestra.run.smithi082.stdout:68719476738 2024-09-10T19:43:00.531 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476739 got 68719476738 for osd.6 2024-09-10T19:43:00.540 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476739 got 68719476738 for osd.2 2024-09-10T19:43:00.549 INFO:teuthology.orchestra.run.smithi082.stdout:64424509443 2024-09-10T19:43:00.561 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509443 for osd.0 2024-09-10T19:43:00.604 INFO:teuthology.orchestra.run.smithi082.stdout:73014444033 2024-09-10T19:43:00.605 INFO:teuthology.orchestra.run.smithi082.stdout:73014444034 2024-09-10T19:43:00.617 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444035 got 73014444033 for osd.7 2024-09-10T19:43:00.618 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444034 for osd.9 2024-09-10T19:43:01.483 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-10T19:43:01.488 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-10T19:43:01.508 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-10T19:43:01.510 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-10T19:43:01.528 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-10T19:43:01.531 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-10T19:43:01.541 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-10T19:43:01.561 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-10T19:43:01.618 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-10T19:43:01.618 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-10T19:43:02.075 INFO:teuthology.orchestra.run.smithi082.stdout:73014444038 2024-09-10T19:43:02.094 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444037 got 73014444038 for osd.5 2024-09-10T19:43:02.094 DEBUG:teuthology.parallel:result is None 2024-09-10T19:43:02.128 INFO:teuthology.orchestra.run.smithi082.stdout:68719476739 2024-09-10T19:43:02.167 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476739 got 68719476739 for osd.3 2024-09-10T19:43:02.167 DEBUG:teuthology.parallel:result is None 2024-09-10T19:43:02.170 INFO:teuthology.orchestra.run.smithi082.stdout:64424509445 2024-09-10T19:43:02.187 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509445 for osd.8 2024-09-10T19:43:02.187 DEBUG:teuthology.parallel:result is None 2024-09-10T19:43:02.216 INFO:teuthology.orchestra.run.smithi082.stdout:64424509444 2024-09-10T19:43:02.230 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509444 for osd.0 2024-09-10T19:43:02.230 DEBUG:teuthology.parallel:result is None 2024-09-10T19:43:02.235 INFO:teuthology.orchestra.run.smithi082.stdout:68719476739 2024-09-10T19:43:02.249 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476739 got 68719476739 for osd.2 2024-09-10T19:43:02.249 DEBUG:teuthology.parallel:result is None 2024-09-10T19:43:02.265 INFO:teuthology.orchestra.run.smithi082.stdout:73014444035 2024-09-10T19:43:02.267 INFO:teuthology.orchestra.run.smithi082.stdout:68719476740 2024-09-10T19:43:02.284 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444035 got 73014444035 for osd.1 2024-09-10T19:43:02.284 DEBUG:teuthology.parallel:result is None 2024-09-10T19:43:02.286 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476740 got 68719476740 for osd.4 2024-09-10T19:43:02.286 DEBUG:teuthology.parallel:result is None 2024-09-10T19:43:02.299 INFO:teuthology.orchestra.run.smithi082.stdout:73014444035 2024-09-10T19:43:02.303 INFO:teuthology.orchestra.run.smithi082.stdout:68719476739 2024-09-10T19:43:02.313 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444035 got 73014444035 for osd.7 2024-09-10T19:43:02.313 DEBUG:teuthology.parallel:result is None 2024-09-10T19:43:02.315 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476739 got 68719476739 for osd.6 2024-09-10T19:43:02.316 DEBUG:teuthology.parallel:result is None 2024-09-10T19:43:02.356 INFO:teuthology.orchestra.run.smithi082.stdout:73014444036 2024-09-10T19:43:02.369 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444036 for osd.9 2024-09-10T19:43:02.369 DEBUG:teuthology.parallel:result is None 2024-09-10T19:43:02.369 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-10T19:43:02.369 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T19:43:02.690 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:43:02.691 INFO:teuthology.orchestra.run.smithi082.stderr:dumped all 2024-09-10T19:43:02.703 INFO:teuthology.orchestra.run.smithi082.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-09-10T19:43:01.008308+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":9,"kb":1048576000,"kb_used":272208,"kb_used_data":1896,"kb_used_omap":15,"kb_used_meta":270064,"kb_avail":1048303792,"statfs":{"total":1073741824000,"available":1073463083008,"internally_reserved":0,"allocated":1941504,"data_stored":644366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":276545973},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.767571"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T19:42:58.242298+0000","last_change":"2024-09-10T19:42:58.242523+0000","last_active":"2024-09-10T19:42:58.242298+0000","last_peered":"2024-09-10T19:42:58.242298+0000","last_clean":"2024-09-10T19:42:58.242298+0000","last_became_active":"2024-09-10T19:42:56.573779+0000","last_became_peered":"2024-09-10T19:42:56.573779+0000","last_unstale":"2024-09-10T19:42:58.242298+0000","last_undegraded":"2024-09-10T19:42:58.242298+0000","last_fullsized":"2024-09-10T19:42:58.242298+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:19:19.008404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047593899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T19:42:58.242255+0000","last_change":"2024-09-10T19:42:58.242455+0000","last_active":"2024-09-10T19:42:58.242255+0000","last_peered":"2024-09-10T19:42:58.242255+0000","last_clean":"2024-09-10T19:42:58.242255+0000","last_became_active":"2024-09-10T19:42:56.250759+0000","last_became_peered":"2024-09-10T19:42:56.250759+0000","last_unstale":"2024-09-10T19:42:58.242255+0000","last_undegraded":"2024-09-10T19:42:58.242255+0000","last_fullsized":"2024-09-10T19:42:58.242255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:35:35.266344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041823700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9],"acting":[6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T19:42:59.369949+0000","last_change":"2024-09-10T19:42:56.240580+0000","last_active":"2024-09-10T19:42:59.369949+0000","last_peered":"2024-09-10T19:42:59.369949+0000","last_clean":"2024-09-10T19:42:59.369949+0000","last_became_active":"2024-09-10T19:42:56.240244+0000","last_became_peered":"2024-09-10T19:42:56.240244+0000","last_unstale":"2024-09-10T19:42:59.369949+0000","last_undegraded":"2024-09-10T19:42:59.369949+0000","last_fullsized":"2024-09-10T19:42:59.369949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:19:19.008404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8],"acting":[4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T19:42:59.369881+0000","last_change":"2024-09-10T19:42:56.238401+0000","last_active":"2024-09-10T19:42:59.369881+0000","last_peered":"2024-09-10T19:42:59.369881+0000","last_clean":"2024-09-10T19:42:59.369881+0000","last_became_active":"2024-09-10T19:42:56.238087+0000","last_became_peered":"2024-09-10T19:42:56.238087+0000","last_unstale":"2024-09-10T19:42:59.369881+0000","last_undegraded":"2024-09-10T19:42:59.369881+0000","last_fullsized":"2024-09-10T19:42:59.369881+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:29:54.354116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"19'1","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T19:42:59.369933+0000","last_change":"2024-09-10T19:42:56.278833+0000","last_active":"2024-09-10T19:42:59.369933+0000","last_peered":"2024-09-10T19:42:59.369933+0000","last_clean":"2024-09-10T19:42:59.369933+0000","last_became_active":"2024-09-10T19:42:56.278016+0000","last_became_peered":"2024-09-10T19:42:56.278016+0000","last_unstale":"2024-09-10T19:42:59.369933+0000","last_undegraded":"2024-09-10T19:42:59.369933+0000","last_fullsized":"2024-09-10T19:42:59.369933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:35:35.266344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"21'2","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T19:42:58.244715+0000","last_change":"2024-09-10T19:42:58.244715+0000","last_active":"2024-09-10T19:42:58.244715+0000","last_peered":"2024-09-10T19:42:58.244715+0000","last_clean":"2024-09-10T19:42:58.244715+0000","last_became_active":"2024-09-10T19:42:56.573796+0000","last_became_peered":"2024-09-10T19:42:56.573796+0000","last_unstale":"2024-09-10T19:42:58.244715+0000","last_undegraded":"2024-09-10T19:42:58.244715+0000","last_fullsized":"2024-09-10T19:42:58.244715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:35:35.266344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084709599999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T19:42:59.821079+0000","last_change":"2024-09-10T19:42:59.821298+0000","last_active":"2024-09-10T19:42:59.821079+0000","last_peered":"2024-09-10T19:42:59.821079+0000","last_clean":"2024-09-10T19:42:59.821079+0000","last_became_active":"2024-09-10T19:42:56.238070+0000","last_became_peered":"2024-09-10T19:42:56.238070+0000","last_unstale":"2024-09-10T19:42:59.821079+0000","last_undegraded":"2024-09-10T19:42:59.821079+0000","last_fullsized":"2024-09-10T19:42:59.821079+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:58:04.941053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048965100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T19:42:59.369815+0000","last_change":"2024-09-10T19:42:59.370022+0000","last_active":"2024-09-10T19:42:59.369815+0000","last_peered":"2024-09-10T19:42:59.369815+0000","last_clean":"2024-09-10T19:42:59.369815+0000","last_became_active":"2024-09-10T19:42:56.240341+0000","last_became_peered":"2024-09-10T19:42:56.240341+0000","last_unstale":"2024-09-10T19:42:59.369815+0000","last_undegraded":"2024-09-10T19:42:59.369815+0000","last_fullsized":"2024-09-10T19:42:59.369815+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:30:48.593419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000443783,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":5,"up_from":17,"seq":73014444038,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27224,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830376,"statfs":{"total":107374182400,"available":107346305024,"internally_reserved":0,"allocated":200704,"data_stored":68136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":17,"seq":73014444036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27228,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830372,"statfs":{"total":107374182400,"available":107346300928,"internally_reserved":0,"allocated":200704,"data_stored":68136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":17,"seq":73014444035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27240,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830360,"statfs":{"total":107374182400,"available":107346288640,"internally_reserved":0,"allocated":204800,"data_stored":68155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":17,"seq":73014444035,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27216,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830384,"statfs":{"total":107374182400,"available":107346313216,"internally_reserved":0,"allocated":188416,"data_stored":61936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":16,"seq":68719476739,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27228,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830372,"statfs":{"total":107374182400,"available":107346300928,"internally_reserved":0,"allocated":200704,"data_stored":68136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":16,"seq":68719476740,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27200,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830400,"statfs":{"total":107374182400,"available":107346329600,"internally_reserved":0,"allocated":176128,"data_stored":55820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,6,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":16,"seq":68719476739,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27200,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830400,"statfs":{"total":107374182400,"available":107346329600,"internally_reserved":0,"allocated":176128,"data_stored":55820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,6,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":16,"seq":68719476739,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27212,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830388,"statfs":{"total":107374182400,"available":107346317312,"internally_reserved":0,"allocated":188416,"data_stored":61936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,4,6,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":15,"seq":64424509445,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27224,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830376,"statfs":{"total":107374182400,"available":107346305024,"internally_reserved":0,"allocated":200704,"data_stored":68136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27236,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830364,"statfs":{"total":107374182400,"available":107346292736,"internally_reserved":0,"allocated":204800,"data_stored":68155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:43:02.704 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T19:43:02.987 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:43:02.987 INFO:teuthology.orchestra.run.smithi082.stderr:dumped all 2024-09-10T19:43:02.999 INFO:teuthology.orchestra.run.smithi082.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-09-10T19:43:01.008308+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":9,"kb":1048576000,"kb_used":272208,"kb_used_data":1896,"kb_used_omap":15,"kb_used_meta":270064,"kb_avail":1048303792,"statfs":{"total":1073741824000,"available":1073463083008,"internally_reserved":0,"allocated":1941504,"data_stored":644366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":276545973},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.767571"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T19:42:58.242298+0000","last_change":"2024-09-10T19:42:58.242523+0000","last_active":"2024-09-10T19:42:58.242298+0000","last_peered":"2024-09-10T19:42:58.242298+0000","last_clean":"2024-09-10T19:42:58.242298+0000","last_became_active":"2024-09-10T19:42:56.573779+0000","last_became_peered":"2024-09-10T19:42:56.573779+0000","last_unstale":"2024-09-10T19:42:58.242298+0000","last_undegraded":"2024-09-10T19:42:58.242298+0000","last_fullsized":"2024-09-10T19:42:58.242298+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:19:19.008404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047593899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T19:42:58.242255+0000","last_change":"2024-09-10T19:42:58.242455+0000","last_active":"2024-09-10T19:42:58.242255+0000","last_peered":"2024-09-10T19:42:58.242255+0000","last_clean":"2024-09-10T19:42:58.242255+0000","last_became_active":"2024-09-10T19:42:56.250759+0000","last_became_peered":"2024-09-10T19:42:56.250759+0000","last_unstale":"2024-09-10T19:42:58.242255+0000","last_undegraded":"2024-09-10T19:42:58.242255+0000","last_fullsized":"2024-09-10T19:42:58.242255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:35:35.266344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041823700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9],"acting":[6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T19:42:59.369949+0000","last_change":"2024-09-10T19:42:56.240580+0000","last_active":"2024-09-10T19:42:59.369949+0000","last_peered":"2024-09-10T19:42:59.369949+0000","last_clean":"2024-09-10T19:42:59.369949+0000","last_became_active":"2024-09-10T19:42:56.240244+0000","last_became_peered":"2024-09-10T19:42:56.240244+0000","last_unstale":"2024-09-10T19:42:59.369949+0000","last_undegraded":"2024-09-10T19:42:59.369949+0000","last_fullsized":"2024-09-10T19:42:59.369949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:19:19.008404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8],"acting":[4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T19:42:59.369881+0000","last_change":"2024-09-10T19:42:56.238401+0000","last_active":"2024-09-10T19:42:59.369881+0000","last_peered":"2024-09-10T19:42:59.369881+0000","last_clean":"2024-09-10T19:42:59.369881+0000","last_became_active":"2024-09-10T19:42:56.238087+0000","last_became_peered":"2024-09-10T19:42:56.238087+0000","last_unstale":"2024-09-10T19:42:59.369881+0000","last_undegraded":"2024-09-10T19:42:59.369881+0000","last_fullsized":"2024-09-10T19:42:59.369881+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:29:54.354116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"19'1","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T19:42:59.369933+0000","last_change":"2024-09-10T19:42:56.278833+0000","last_active":"2024-09-10T19:42:59.369933+0000","last_peered":"2024-09-10T19:42:59.369933+0000","last_clean":"2024-09-10T19:42:59.369933+0000","last_became_active":"2024-09-10T19:42:56.278016+0000","last_became_peered":"2024-09-10T19:42:56.278016+0000","last_unstale":"2024-09-10T19:42:59.369933+0000","last_undegraded":"2024-09-10T19:42:59.369933+0000","last_fullsized":"2024-09-10T19:42:59.369933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:35:35.266344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"21'2","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T19:42:58.244715+0000","last_change":"2024-09-10T19:42:58.244715+0000","last_active":"2024-09-10T19:42:58.244715+0000","last_peered":"2024-09-10T19:42:58.244715+0000","last_clean":"2024-09-10T19:42:58.244715+0000","last_became_active":"2024-09-10T19:42:56.573796+0000","last_became_peered":"2024-09-10T19:42:56.573796+0000","last_unstale":"2024-09-10T19:42:58.244715+0000","last_undegraded":"2024-09-10T19:42:58.244715+0000","last_fullsized":"2024-09-10T19:42:58.244715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:35:35.266344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084709599999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T19:42:59.821079+0000","last_change":"2024-09-10T19:42:59.821298+0000","last_active":"2024-09-10T19:42:59.821079+0000","last_peered":"2024-09-10T19:42:59.821079+0000","last_clean":"2024-09-10T19:42:59.821079+0000","last_became_active":"2024-09-10T19:42:56.238070+0000","last_became_peered":"2024-09-10T19:42:56.238070+0000","last_unstale":"2024-09-10T19:42:59.821079+0000","last_undegraded":"2024-09-10T19:42:59.821079+0000","last_fullsized":"2024-09-10T19:42:59.821079+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:58:04.941053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048965100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-10T19:42:59.369815+0000","last_change":"2024-09-10T19:42:59.370022+0000","last_active":"2024-09-10T19:42:59.369815+0000","last_peered":"2024-09-10T19:42:59.369815+0000","last_clean":"2024-09-10T19:42:59.369815+0000","last_became_active":"2024-09-10T19:42:56.240341+0000","last_became_peered":"2024-09-10T19:42:56.240341+0000","last_unstale":"2024-09-10T19:42:59.369815+0000","last_undegraded":"2024-09-10T19:42:59.369815+0000","last_fullsized":"2024-09-10T19:42:59.369815+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:30:48.593419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000443783,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":5,"up_from":17,"seq":73014444038,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27224,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830376,"statfs":{"total":107374182400,"available":107346305024,"internally_reserved":0,"allocated":200704,"data_stored":68136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":17,"seq":73014444036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27228,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830372,"statfs":{"total":107374182400,"available":107346300928,"internally_reserved":0,"allocated":200704,"data_stored":68136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":17,"seq":73014444035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27240,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830360,"statfs":{"total":107374182400,"available":107346288640,"internally_reserved":0,"allocated":204800,"data_stored":68155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":17,"seq":73014444035,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27216,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830384,"statfs":{"total":107374182400,"available":107346313216,"internally_reserved":0,"allocated":188416,"data_stored":61936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":16,"seq":68719476739,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27228,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830372,"statfs":{"total":107374182400,"available":107346300928,"internally_reserved":0,"allocated":200704,"data_stored":68136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":16,"seq":68719476740,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27200,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830400,"statfs":{"total":107374182400,"available":107346329600,"internally_reserved":0,"allocated":176128,"data_stored":55820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,6,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":16,"seq":68719476739,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27200,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830400,"statfs":{"total":107374182400,"available":107346329600,"internally_reserved":0,"allocated":176128,"data_stored":55820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,6,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":16,"seq":68719476739,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27212,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830388,"statfs":{"total":107374182400,"available":107346317312,"internally_reserved":0,"allocated":188416,"data_stored":61936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,4,6,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":15,"seq":64424509445,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27224,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830376,"statfs":{"total":107374182400,"available":107346305024,"internally_reserved":0,"allocated":200704,"data_stored":68136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27236,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830364,"statfs":{"total":107374182400,"available":107346292736,"internally_reserved":0,"allocated":204800,"data_stored":68155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:43:03.000 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-10T19:43:03.000 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-10T19:43:03.000 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-10T19:43:03.000 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-10T19:43:03.317 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:43:03.317 INFO:teuthology.orchestra.run.smithi082.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-10T19:43:03.329 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-10T19:43:03.330 INFO:teuthology.run_tasks:Running task workunit... 2024-09-10T19:43:03.344 INFO:tasks.workunit:Pulling workunits from ref bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T19:43:03.344 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-10T19:43:03.344 INFO:tasks.workunit:timeout=3h 2024-09-10T19:43:03.344 INFO:tasks.workunit:cleanup=True 2024-09-10T19:43:03.345 DEBUG:teuthology.orchestra.run.smithi082:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-10T19:43:03.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:43:03.360 INFO:teuthology.orchestra.run.smithi082.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-10T19:43:03.360 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-10T19:43:03.415 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-10T19:43:03.415 DEBUG:teuthology.orchestra.run.smithi082:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-10T19:43:03.470 DEBUG:teuthology.orchestra.run.smithi082:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T19:43:03.526 INFO:tasks.workunit.client.0.smithi082.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-10T19:43:53.664 INFO:tasks.workunit.client.0.smithi082.stderr:Note: switching to 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'. 2024-09-10T19:43:53.664 INFO:tasks.workunit.client.0.smithi082.stderr: 2024-09-10T19:43:53.664 INFO:tasks.workunit.client.0.smithi082.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-10T19:43:53.664 INFO:tasks.workunit.client.0.smithi082.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-10T19:43:53.664 INFO:tasks.workunit.client.0.smithi082.stderr:state without impacting any branches by switching back to a branch. 2024-09-10T19:43:53.664 INFO:tasks.workunit.client.0.smithi082.stderr: 2024-09-10T19:43:53.664 INFO:tasks.workunit.client.0.smithi082.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-10T19:43:53.665 INFO:tasks.workunit.client.0.smithi082.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-10T19:43:53.665 INFO:tasks.workunit.client.0.smithi082.stderr: 2024-09-10T19:43:53.665 INFO:tasks.workunit.client.0.smithi082.stderr: git switch -c 2024-09-10T19:43:53.665 INFO:tasks.workunit.client.0.smithi082.stderr: 2024-09-10T19:43:53.665 INFO:tasks.workunit.client.0.smithi082.stderr:Or undo this operation with: 2024-09-10T19:43:53.665 INFO:tasks.workunit.client.0.smithi082.stderr: 2024-09-10T19:43:53.665 INFO:tasks.workunit.client.0.smithi082.stderr: git switch - 2024-09-10T19:43:53.665 INFO:tasks.workunit.client.0.smithi082.stderr: 2024-09-10T19:43:53.665 INFO:tasks.workunit.client.0.smithi082.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-10T19:43:53.665 INFO:tasks.workunit.client.0.smithi082.stderr: 2024-09-10T19:43:53.665 INFO:tasks.workunit.client.0.smithi082.stderr:HEAD is now at bad1f558c32 Merge branch 'wip-ifed-bluestore-tool-really-really' of https://github.com/ifed01/ceph into wip-bharath7-testing-2024-09-10-1409 2024-09-10T19:43:53.667 DEBUG:teuthology.orchestra.run.smithi082:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-10T19:43:53.735 INFO:tasks.workunit.client.0.smithi082.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-10T19:43:53.737 INFO:tasks.workunit.client.0.smithi082.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-10T19:43:53.737 INFO:tasks.workunit.client.0.smithi082.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-10T19:43:54.330 INFO:tasks.workunit.client.0.smithi082.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-10T19:43:54.403 INFO:tasks.workunit.client.0.smithi082.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-10T19:43:54.443 INFO:tasks.workunit.client.0.smithi082.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-10T19:43:54.445 INFO:tasks.workunit.client.0.smithi082.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-10T19:43:54.445 INFO:tasks.workunit.client.0.smithi082.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-10T19:43:54.484 INFO:tasks.workunit.client.0.smithi082.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-10T19:43:54.487 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-10T19:43:54.487 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-10T19:43:54.543 INFO:tasks.workunit:Running workunits matching rados/test_health_warnings.sh on client.0... 2024-09-10T19:43:54.545 INFO:tasks.workunit:Running workunit rados/test_health_warnings.sh... 2024-09-10T19:43:54.545 DEBUG:teuthology.orchestra.run.smithi082:workunit test rados/test_health_warnings.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_health_warnings.sh 2024-09-10T19:43:54.603 INFO:tasks.workunit.client.0.smithi082.stderr:+ crushtool -o crushmap --build --num_osds 10 host straw 2 rack straw 2 row straw 2 root straw 0 2024-09-10T19:43:54.619 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph osd setcrushmap -i crushmap 2024-09-10T19:43:56.447 INFO:tasks.workunit.client.0.smithi082.stderr:7 2024-09-10T19:43:56.467 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph osd tree 2024-09-10T19:43:56.846 INFO:tasks.workunit.client.0.smithi082.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-10T19:43:56.846 INFO:tasks.workunit.client.0.smithi082.stdout:-11 10.00000 root root 2024-09-10T19:43:56.847 INFO:tasks.workunit.client.0.smithi082.stdout: -9 8.00000 row row0 2024-09-10T19:43:56.847 INFO:tasks.workunit.client.0.smithi082.stdout: -6 4.00000 rack rack0 2024-09-10T19:43:56.847 INFO:tasks.workunit.client.0.smithi082.stdout: -1 2.00000 host host0 2024-09-10T19:43:56.847 INFO:tasks.workunit.client.0.smithi082.stdout: 0 1.00000 osd.0 up 1.00000 1.00000 2024-09-10T19:43:56.847 INFO:tasks.workunit.client.0.smithi082.stdout: 1 1.00000 osd.1 up 1.00000 1.00000 2024-09-10T19:43:56.847 INFO:tasks.workunit.client.0.smithi082.stdout: -2 2.00000 host host1 2024-09-10T19:43:56.847 INFO:tasks.workunit.client.0.smithi082.stdout: 2 1.00000 osd.2 up 1.00000 1.00000 2024-09-10T19:43:56.847 INFO:tasks.workunit.client.0.smithi082.stdout: 3 1.00000 osd.3 up 1.00000 1.00000 2024-09-10T19:43:56.847 INFO:tasks.workunit.client.0.smithi082.stdout: -7 4.00000 rack rack1 2024-09-10T19:43:56.847 INFO:tasks.workunit.client.0.smithi082.stdout: -3 2.00000 host host2 2024-09-10T19:43:56.847 INFO:tasks.workunit.client.0.smithi082.stdout: 4 1.00000 osd.4 up 1.00000 1.00000 2024-09-10T19:43:56.847 INFO:tasks.workunit.client.0.smithi082.stdout: 5 1.00000 osd.5 up 1.00000 1.00000 2024-09-10T19:43:56.848 INFO:tasks.workunit.client.0.smithi082.stdout: -4 2.00000 host host3 2024-09-10T19:43:56.848 INFO:tasks.workunit.client.0.smithi082.stdout: 6 1.00000 osd.6 up 1.00000 1.00000 2024-09-10T19:43:56.848 INFO:tasks.workunit.client.0.smithi082.stdout: 7 1.00000 osd.7 up 1.00000 1.00000 2024-09-10T19:43:56.848 INFO:tasks.workunit.client.0.smithi082.stdout:-10 2.00000 row row1 2024-09-10T19:43:56.848 INFO:tasks.workunit.client.0.smithi082.stdout: -8 2.00000 rack rack2 2024-09-10T19:43:56.848 INFO:tasks.workunit.client.0.smithi082.stdout: -5 2.00000 host host4 2024-09-10T19:43:56.848 INFO:tasks.workunit.client.0.smithi082.stdout: 8 1.00000 osd.8 up 1.00000 1.00000 2024-09-10T19:43:56.848 INFO:tasks.workunit.client.0.smithi082.stdout: 9 1.00000 osd.9 up 1.00000 1.00000 2024-09-10T19:43:56.857 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph tell 'osd.*' injectargs --osd_max_markdown_count 1024 --osd_max_markdown_period 1 2024-09-10T19:43:56.970 INFO:tasks.workunit.client.0.smithi082.stdout:osd.0: {} 2024-09-10T19:43:56.999 INFO:tasks.workunit.client.0.smithi082.stdout:osd.1: {} 2024-09-10T19:43:57.028 INFO:tasks.workunit.client.0.smithi082.stdout:osd.2: {} 2024-09-10T19:43:57.056 INFO:tasks.workunit.client.0.smithi082.stdout:osd.3: {} 2024-09-10T19:43:57.083 INFO:tasks.workunit.client.0.smithi082.stdout:osd.4: {} 2024-09-10T19:43:57.111 INFO:tasks.workunit.client.0.smithi082.stdout:osd.5: {} 2024-09-10T19:43:57.139 INFO:tasks.workunit.client.0.smithi082.stdout:osd.6: {} 2024-09-10T19:43:57.168 INFO:tasks.workunit.client.0.smithi082.stdout:osd.7: {} 2024-09-10T19:43:57.195 INFO:tasks.workunit.client.0.smithi082.stdout:osd.8: {} 2024-09-10T19:43:57.223 INFO:tasks.workunit.client.0.smithi082.stdout:osd.9: {} 2024-09-10T19:43:57.233 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph osd set noout 2024-09-10T19:43:59.468 INFO:tasks.workunit.client.0.smithi082.stderr:noout is set 2024-09-10T19:43:59.483 INFO:tasks.workunit.client.0.smithi082.stderr:+ test_mark_two_osds_same_host_down 2024-09-10T19:43:59.483 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph osd set noup 2024-09-10T19:44:01.474 INFO:tasks.workunit.client.0.smithi082.stderr:noup is set 2024-09-10T19:44:01.488 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph osd down osd.0 osd.1 2024-09-10T19:44:01.766 INFO:tasks.ceph.mon.a.smithi082.stderr:2024-09-10T19:44:01.763+0000 7f750832a640 -1 mon.a@0(leader).osd e28 definitely_dead 0 2024-09-10T19:44:02.595 INFO:tasks.ceph.mon.a.smithi082.stderr:2024-09-10T19:44:02.593+0000 7f750832a640 -1 mon.a@0(leader).osd e29 definitely_dead 0 2024-09-10T19:44:02.595 INFO:tasks.workunit.client.0.smithi082.stderr:osd.0 is already down. osd.1 is already down. 2024-09-10T19:44:02.607 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph health detail 2024-09-10T19:44:03.108 INFO:tasks.workunit.client.0.smithi082.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2024-09-10T19:44:03.109 INFO:tasks.workunit.client.0.smithi082.stdout:[WRN] OSDMAP_FLAGS: noup,noout flag(s) set 2024-09-10T19:44:03.109 INFO:tasks.workunit.client.0.smithi082.stdout:[WRN] OSD_DOWN: 2 osds down 2024-09-10T19:44:03.109 INFO:tasks.workunit.client.0.smithi082.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2024-09-10T19:44:03.109 INFO:tasks.workunit.client.0.smithi082.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2024-09-10T19:44:03.109 INFO:tasks.workunit.client.0.smithi082.stdout:[WRN] OSD_HOST_DOWN: 1 host (2 osds) down 2024-09-10T19:44:03.109 INFO:tasks.workunit.client.0.smithi082.stdout: host host0 (root=root,row=row0,rack=rack0) (2 osds) is down 2024-09-10T19:44:03.120 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph health 2024-09-10T19:44:03.120 INFO:tasks.workunit.client.0.smithi082.stderr:+ grep '1 host' 2024-09-10T19:44:03.239 INFO:tasks.ceph.osd.1.smithi082.stderr:2024-09-10T19:44:03.235+0000 7fcea7721640 -1 osd.1 29 osdmap NOUP flag is set, waiting for it to clear 2024-09-10T19:44:03.479 INFO:tasks.ceph.osd.1.smithi082.stderr:2024-09-10T19:44:03.476+0000 7fce96d02640 -1 osd.1 30 osdmap NOUP flag is set, waiting for it to clear 2024-09-10T19:44:03.574 INFO:tasks.workunit.client.0.smithi082.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2024-09-10T19:44:03.574 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph health 2024-09-10T19:44:03.574 INFO:tasks.workunit.client.0.smithi082.stderr:+ grep '2 osds' 2024-09-10T19:44:03.798 INFO:tasks.ceph.osd.0.smithi082.stderr:2024-09-10T19:44:03.795+0000 7fdeca7d2640 -1 osd.0 30 osdmap NOUP flag is set, waiting for it to clear 2024-09-10T19:44:04.021 INFO:tasks.workunit.client.0.smithi082.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2024-09-10T19:44:04.021 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph health detail 2024-09-10T19:44:04.022 INFO:tasks.workunit.client.0.smithi082.stderr:+ grep osd.0 2024-09-10T19:44:04.478 INFO:tasks.ceph.osd.0.smithi082.stderr:2024-09-10T19:44:04.475+0000 7fdebddbb640 -1 osd.0 31 osdmap NOUP flag is set, waiting for it to clear 2024-09-10T19:44:04.478 INFO:tasks.ceph.osd.1.smithi082.stderr:2024-09-10T19:44:04.475+0000 7fce96d02640 -1 osd.1 31 osdmap NOUP flag is set, waiting for it to clear 2024-09-10T19:44:04.537 INFO:tasks.workunit.client.0.smithi082.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2024-09-10T19:44:04.538 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph health detail 2024-09-10T19:44:04.538 INFO:tasks.workunit.client.0.smithi082.stderr:+ grep osd.1 2024-09-10T19:44:05.049 INFO:tasks.workunit.client.0.smithi082.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2024-09-10T19:44:05.049 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph osd unset noup 2024-09-10T19:44:05.498 INFO:tasks.ceph.osd.1.smithi082.stderr:2024-09-10T19:44:05.495+0000 7fce9ed2c640 -1 osd.1 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:44:05.498 INFO:tasks.ceph.osd.0.smithi082.stderr:2024-09-10T19:44:05.495+0000 7fdec5de5640 -1 osd.0 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:44:06.495 INFO:tasks.workunit.client.0.smithi082.stderr:noup is unset 2024-09-10T19:44:06.511 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait_for_healthy 2024-09-10T19:44:06.511 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph health 2024-09-10T19:44:06.511 INFO:tasks.workunit.client.0.smithi082.stderr:+ grep down 2024-09-10T19:44:06.961 INFO:tasks.workunit.client.0.smithi082.stderr:+ test_mark_two_osds_same_rack_down 2024-09-10T19:44:06.961 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph osd set noup 2024-09-10T19:44:08.503 INFO:tasks.workunit.client.0.smithi082.stderr:noup is set 2024-09-10T19:44:08.517 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph osd down osd.8 osd.9 2024-09-10T19:44:08.793 INFO:tasks.ceph.mon.a.smithi082.stderr:2024-09-10T19:44:08.791+0000 7f750832a640 -1 mon.a@0(leader).osd e35 definitely_dead 0 2024-09-10T19:44:09.625 INFO:tasks.ceph.mon.a.smithi082.stderr:2024-09-10T19:44:09.623+0000 7f750832a640 -1 mon.a@0(leader).osd e36 definitely_dead 0 2024-09-10T19:44:09.626 INFO:tasks.workunit.client.0.smithi082.stderr:osd.8 is already down. osd.9 is already down. 2024-09-10T19:44:09.638 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph health detail 2024-09-10T19:44:10.138 INFO:tasks.workunit.client.0.smithi082.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down 2024-09-10T19:44:10.138 INFO:tasks.workunit.client.0.smithi082.stdout:[WRN] OSDMAP_FLAGS: noup,noout flag(s) set 2024-09-10T19:44:10.138 INFO:tasks.workunit.client.0.smithi082.stdout:[WRN] OSD_DOWN: 2 osds down 2024-09-10T19:44:10.138 INFO:tasks.workunit.client.0.smithi082.stdout: osd.8 (root=root,row=row1,rack=rack2,host=host4) is down 2024-09-10T19:44:10.138 INFO:tasks.workunit.client.0.smithi082.stdout: osd.9 (root=root,row=row1,rack=rack2,host=host4) is down 2024-09-10T19:44:10.139 INFO:tasks.workunit.client.0.smithi082.stdout:[WRN] OSD_HOST_DOWN: 1 host (2 osds) down 2024-09-10T19:44:10.139 INFO:tasks.workunit.client.0.smithi082.stdout: host host4 (root=root,row=row1,rack=rack2) (2 osds) is down 2024-09-10T19:44:10.139 INFO:tasks.workunit.client.0.smithi082.stdout:[WRN] OSD_RACK_DOWN: 1 rack (2 osds) down 2024-09-10T19:44:10.139 INFO:tasks.workunit.client.0.smithi082.stdout: rack rack2 (root=root,row=row1) (2 osds) is down 2024-09-10T19:44:10.139 INFO:tasks.workunit.client.0.smithi082.stdout:[WRN] OSD_ROW_DOWN: 1 row (2 osds) down 2024-09-10T19:44:10.139 INFO:tasks.workunit.client.0.smithi082.stdout: row row1 (root=root) (2 osds) is down 2024-09-10T19:44:10.149 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph health 2024-09-10T19:44:10.149 INFO:tasks.workunit.client.0.smithi082.stderr:+ grep '1 host' 2024-09-10T19:44:10.310 INFO:tasks.ceph.osd.8.smithi082.stderr:2024-09-10T19:44:10.308+0000 7f61557b3640 -1 osd.8 36 osdmap NOUP flag is set, waiting for it to clear 2024-09-10T19:44:10.511 INFO:tasks.ceph.osd.8.smithi082.stderr:2024-09-10T19:44:10.508+0000 7f61455a8640 -1 osd.8 37 osdmap NOUP flag is set, waiting for it to clear 2024-09-10T19:44:10.605 INFO:tasks.workunit.client.0.smithi082.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down 2024-09-10T19:44:10.606 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph health 2024-09-10T19:44:10.606 INFO:tasks.workunit.client.0.smithi082.stderr:+ grep '1 rack' 2024-09-10T19:44:11.058 INFO:tasks.workunit.client.0.smithi082.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down 2024-09-10T19:44:11.058 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph health 2024-09-10T19:44:11.058 INFO:tasks.workunit.client.0.smithi082.stderr:+ grep '1 row' 2024-09-10T19:44:11.505 INFO:tasks.workunit.client.0.smithi082.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down 2024-09-10T19:44:11.506 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph health 2024-09-10T19:44:11.506 INFO:tasks.workunit.client.0.smithi082.stderr:+ grep '2 osds' 2024-09-10T19:44:11.956 INFO:tasks.workunit.client.0.smithi082.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down 2024-09-10T19:44:11.957 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph health detail 2024-09-10T19:44:11.957 INFO:tasks.workunit.client.0.smithi082.stderr:+ grep osd.8 2024-09-10T19:44:12.468 INFO:tasks.workunit.client.0.smithi082.stdout: osd.8 (root=root,row=row1,rack=rack2,host=host4) is down 2024-09-10T19:44:12.469 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph health detail 2024-09-10T19:44:12.469 INFO:tasks.workunit.client.0.smithi082.stderr:+ grep osd.9 2024-09-10T19:44:12.982 INFO:tasks.workunit.client.0.smithi082.stdout: osd.9 (root=root,row=row1,rack=rack2,host=host4) is down 2024-09-10T19:44:12.982 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph osd unset noup 2024-09-10T19:44:13.521 INFO:tasks.ceph.osd.8.smithi082.stderr:2024-09-10T19:44:13.519+0000 7f614d5d2640 -1 osd.8 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:44:14.328 INFO:tasks.ceph.osd.9.smithi082.stderr:2024-09-10T19:44:14.326+0000 7fa15404a640 -1 osd.9 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:44:14.518 INFO:tasks.workunit.client.0.smithi082.stderr:noup is unset 2024-09-10T19:44:14.534 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait_for_healthy 2024-09-10T19:44:14.534 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph health 2024-09-10T19:44:14.534 INFO:tasks.workunit.client.0.smithi082.stderr:+ grep down 2024-09-10T19:44:14.988 INFO:tasks.workunit.client.0.smithi082.stderr:+ test_mark_all_but_last_osds_down 2024-09-10T19:44:14.988 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph osd set noup 2024-09-10T19:44:16.532 INFO:tasks.workunit.client.0.smithi082.stderr:noup is set 2024-09-10T19:44:16.547 INFO:tasks.workunit.client.0.smithi082.stderr:++ ceph osd ls 2024-09-10T19:44:16.547 INFO:tasks.workunit.client.0.smithi082.stderr:++ sed '$d' 2024-09-10T19:44:16.930 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph osd down 0 1 2 3 4 5 6 7 8 2024-09-10T19:44:17.210 INFO:tasks.ceph.mon.a.smithi082.stderr:2024-09-10T19:44:17.208+0000 7f750832a640 -1 mon.a@0(leader).osd e41 definitely_dead 0 2024-09-10T19:44:17.649 INFO:tasks.ceph.mon.a.smithi082.stderr:2024-09-10T19:44:17.646+0000 7f750832a640 -1 mon.a@0(leader).osd e42 definitely_dead 0 2024-09-10T19:44:17.649 INFO:tasks.workunit.client.0.smithi082.stderr:osd.0 is already down. osd.1 is already down. osd.2 is already down. osd.3 is already down. osd.4 is already down. osd.5 is already down. osd.6 is already down. osd.7 is already down. osd.8 is already down. 2024-09-10T19:44:17.659 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph health detail 2024-09-10T19:44:18.164 INFO:tasks.workunit.client.0.smithi082.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2024-09-10T19:44:18.164 INFO:tasks.workunit.client.0.smithi082.stdout:[WRN] OSDMAP_FLAGS: noup,noout flag(s) set 2024-09-10T19:44:18.164 INFO:tasks.workunit.client.0.smithi082.stdout:[WRN] OSD_DOWN: 9 osds down 2024-09-10T19:44:18.164 INFO:tasks.workunit.client.0.smithi082.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2024-09-10T19:44:18.164 INFO:tasks.workunit.client.0.smithi082.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2024-09-10T19:44:18.164 INFO:tasks.workunit.client.0.smithi082.stdout: osd.2 (root=root,row=row0,rack=rack0,host=host1) is down 2024-09-10T19:44:18.165 INFO:tasks.workunit.client.0.smithi082.stdout: osd.3 (root=root,row=row0,rack=rack0,host=host1) is down 2024-09-10T19:44:18.165 INFO:tasks.workunit.client.0.smithi082.stdout: osd.4 (root=root,row=row0,rack=rack1,host=host2) is down 2024-09-10T19:44:18.165 INFO:tasks.workunit.client.0.smithi082.stdout: osd.5 (root=root,row=row0,rack=rack1,host=host2) is down 2024-09-10T19:44:18.165 INFO:tasks.workunit.client.0.smithi082.stdout: osd.6 (root=root,row=row0,rack=rack1,host=host3) is down 2024-09-10T19:44:18.165 INFO:tasks.workunit.client.0.smithi082.stdout: osd.7 (root=root,row=row0,rack=rack1,host=host3) is down 2024-09-10T19:44:18.165 INFO:tasks.workunit.client.0.smithi082.stdout: osd.8 (root=root,row=row1,rack=rack2,host=host4) is down 2024-09-10T19:44:18.165 INFO:tasks.workunit.client.0.smithi082.stdout:[WRN] OSD_HOST_DOWN: 4 hosts (8 osds) down 2024-09-10T19:44:18.165 INFO:tasks.workunit.client.0.smithi082.stdout: host host0 (root=root,row=row0,rack=rack0) (2 osds) is down 2024-09-10T19:44:18.165 INFO:tasks.workunit.client.0.smithi082.stdout: host host1 (root=root,row=row0,rack=rack0) (2 osds) is down 2024-09-10T19:44:18.165 INFO:tasks.workunit.client.0.smithi082.stdout: host host2 (root=root,row=row0,rack=rack1) (2 osds) is down 2024-09-10T19:44:18.165 INFO:tasks.workunit.client.0.smithi082.stdout: host host3 (root=root,row=row0,rack=rack1) (2 osds) is down 2024-09-10T19:44:18.165 INFO:tasks.workunit.client.0.smithi082.stdout:[WRN] OSD_RACK_DOWN: 2 racks (8 osds) down 2024-09-10T19:44:18.165 INFO:tasks.workunit.client.0.smithi082.stdout: rack rack0 (root=root,row=row0) (4 osds) is down 2024-09-10T19:44:18.166 INFO:tasks.workunit.client.0.smithi082.stdout: rack rack1 (root=root,row=row0) (4 osds) is down 2024-09-10T19:44:18.166 INFO:tasks.workunit.client.0.smithi082.stdout:[WRN] OSD_ROW_DOWN: 1 row (8 osds) down 2024-09-10T19:44:18.166 INFO:tasks.workunit.client.0.smithi082.stdout: row row0 (root=root) (8 osds) is down 2024-09-10T19:44:18.176 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph health 2024-09-10T19:44:18.176 INFO:tasks.workunit.client.0.smithi082.stderr:+ grep '1 row' 2024-09-10T19:44:18.624 INFO:tasks.workunit.client.0.smithi082.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2024-09-10T19:44:18.624 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph health 2024-09-10T19:44:18.625 INFO:tasks.workunit.client.0.smithi082.stderr:+ grep '2 racks' 2024-09-10T19:44:18.728 INFO:tasks.ceph.osd.0.smithi082.stderr:2024-09-10T19:44:18.726+0000 7fdec9fbe640 -1 osd.0 43 osdmap NOUP flag is set, waiting for it to clear 2024-09-10T19:44:18.905 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-10T19:44:18.903+0000 7fc44f78e640 -1 osd.5 43 osdmap NOUP flag is set, waiting for it to clear 2024-09-10T19:44:19.085 INFO:tasks.workunit.client.0.smithi082.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2024-09-10T19:44:19.086 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph health 2024-09-10T19:44:19.086 INFO:tasks.workunit.client.0.smithi082.stderr:+ grep '4 hosts' 2024-09-10T19:44:19.107 INFO:tasks.ceph.osd.1.smithi082.stderr:2024-09-10T19:44:19.105+0000 7fcea6f0d640 -1 osd.1 43 osdmap NOUP flag is set, waiting for it to clear 2024-09-10T19:44:19.537 INFO:tasks.workunit.client.0.smithi082.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2024-09-10T19:44:19.538 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph health 2024-09-10T19:44:19.538 INFO:tasks.workunit.client.0.smithi082.stderr:+ grep '9 osds' 2024-09-10T19:44:19.542 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-10T19:44:19.540+0000 7fc43ed78640 -1 osd.5 44 osdmap NOUP flag is set, waiting for it to clear 2024-09-10T19:44:19.543 INFO:tasks.ceph.osd.0.smithi082.stderr:2024-09-10T19:44:19.540+0000 7fdebddbb640 -1 osd.0 44 osdmap NOUP flag is set, waiting for it to clear 2024-09-10T19:44:19.543 INFO:tasks.ceph.osd.1.smithi082.stderr:2024-09-10T19:44:19.540+0000 7fce96d02640 -1 osd.1 44 osdmap NOUP flag is set, waiting for it to clear 2024-09-10T19:44:19.755 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-10T19:44:19.753+0000 7f9ca3d32640 -1 osd.4 44 osdmap NOUP flag is set, waiting for it to clear 2024-09-10T19:44:19.990 INFO:tasks.workunit.client.0.smithi082.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2024-09-10T19:44:19.990 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph osd unset noup 2024-09-10T19:44:20.020 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-09-10T19:44:20.018+0000 7f2476b99640 -1 osd.7 44 osdmap NOUP flag is set, waiting for it to clear 2024-09-10T19:44:20.334 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-10T19:44:20.332+0000 7f0620253640 -1 osd.3 44 osdmap NOUP flag is set, waiting for it to clear 2024-09-10T19:44:20.546 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-09-10T19:44:20.544+0000 7f246e1a4640 -1 osd.7 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:44:20.546 INFO:tasks.ceph.osd.1.smithi082.stderr:2024-09-10T19:44:20.544+0000 7fce9ed2c640 -1 osd.1 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:44:20.547 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-10T19:44:20.544+0000 7fc446d99640 -1 osd.5 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:44:20.547 INFO:tasks.ceph.osd.0.smithi082.stderr:2024-09-10T19:44:20.544+0000 7fdec5de5640 -1 osd.0 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:44:20.547 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-10T19:44:20.545+0000 7f061b866640 -1 osd.3 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:44:20.547 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-10T19:44:20.545+0000 7f9c9f345640 -1 osd.4 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:44:21.034 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-09-10T19:44:21.032+0000 7f6d3f44b640 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:44:21.375 INFO:tasks.ceph.osd.8.smithi082.stderr:2024-09-10T19:44:21.373+0000 7f614d5d2640 -1 osd.8 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:44:21.546 INFO:tasks.workunit.client.0.smithi082.stderr:noup is unset 2024-09-10T19:44:21.578 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait_for_healthy 2024-09-10T19:44:21.578 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph health 2024-09-10T19:44:21.578 INFO:tasks.workunit.client.0.smithi082.stderr:+ grep down 2024-09-10T19:44:22.035 INFO:tasks.workunit.client.0.smithi082.stdout:HEALTH_WARN noout flag(s) set; 1 osds down 2024-09-10T19:44:22.035 INFO:tasks.workunit.client.0.smithi082.stderr:+ sleep 1 2024-09-10T19:44:22.565 INFO:tasks.ceph.osd.2.smithi082.stderr:2024-09-10T19:44:22.563+0000 7fae891b9640 -1 osd.2 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:44:23.037 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph health 2024-09-10T19:44:23.037 INFO:tasks.workunit.client.0.smithi082.stderr:+ grep down 2024-09-10T19:44:23.500 INFO:tasks.workunit.client.0.smithi082.stdout:HEALTH_WARN noout flag(s) set; 1 osds down 2024-09-10T19:44:23.500 INFO:tasks.workunit.client.0.smithi082.stderr:+ sleep 1 2024-09-10T19:44:24.502 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph health 2024-09-10T19:44:24.502 INFO:tasks.workunit.client.0.smithi082.stderr:+ grep down 2024-09-10T19:44:24.966 INFO:tasks.workunit.client.0.smithi082.stderr:+ test_mark_two_osds_same_host_down_with_classes 2024-09-10T19:44:24.966 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph osd set noup 2024-09-10T19:44:26.568 INFO:tasks.workunit.client.0.smithi082.stderr:noup is set 2024-09-10T19:44:26.583 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph osd crush set-device-class ssd osd.0 osd.2 osd.4 osd.6 osd.8 2024-09-10T19:44:28.575 INFO:tasks.workunit.client.0.smithi082.stderr:osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. osd.2 already set to class ssd. set-device-class item id 2 name 'osd.2' device_class 'ssd': no change. osd.4 already set to class ssd. set-device-class item id 4 name 'osd.4' device_class 'ssd': no change. osd.6 already set to class ssd. set-device-class item id 6 name 'osd.6' device_class 'ssd': no change. osd.8 already set to class ssd. set-device-class item id 8 name 'osd.8' device_class 'ssd': no change. set osd(s) to class 'ssd' 2024-09-10T19:44:28.590 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph osd crush set-device-class hdd osd.1 osd.3 osd.5 osd.7 osd.9 2024-09-10T19:44:30.585 INFO:tasks.workunit.client.0.smithi082.stderr:osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. osd.3 already set to class hdd. set-device-class item id 3 name 'osd.3' device_class 'hdd': no change. osd.5 already set to class hdd. set-device-class item id 5 name 'osd.5' device_class 'hdd': no change. osd.7 already set to class hdd. set-device-class item id 7 name 'osd.7' device_class 'hdd': no change. osd.9 already set to class hdd. set-device-class item id 9 name 'osd.9' device_class 'hdd': no change. set osd(s) to class 'hdd' 2024-09-10T19:44:30.598 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph osd down osd.0 osd.1 2024-09-10T19:44:30.877 INFO:tasks.ceph.mon.a.smithi082.stderr:2024-09-10T19:44:30.875+0000 7f750832a640 -1 mon.a@0(leader).osd e54 definitely_dead 0 2024-09-10T19:44:31.702 INFO:tasks.ceph.mon.a.smithi082.stderr:2024-09-10T19:44:31.700+0000 7f750832a640 -1 mon.a@0(leader).osd e55 definitely_dead 0 2024-09-10T19:44:31.702 INFO:tasks.workunit.client.0.smithi082.stderr:osd.0 is already down. osd.1 is already down. 2024-09-10T19:44:31.713 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph health detail 2024-09-10T19:44:32.218 INFO:tasks.workunit.client.0.smithi082.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2024-09-10T19:44:32.218 INFO:tasks.workunit.client.0.smithi082.stdout:[WRN] OSDMAP_FLAGS: noup,noout flag(s) set 2024-09-10T19:44:32.218 INFO:tasks.workunit.client.0.smithi082.stdout:[WRN] OSD_DOWN: 2 osds down 2024-09-10T19:44:32.218 INFO:tasks.workunit.client.0.smithi082.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2024-09-10T19:44:32.218 INFO:tasks.workunit.client.0.smithi082.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2024-09-10T19:44:32.219 INFO:tasks.workunit.client.0.smithi082.stdout:[WRN] OSD_HOST_DOWN: 1 host (2 osds) down 2024-09-10T19:44:32.219 INFO:tasks.workunit.client.0.smithi082.stdout: host host0 (root=root,row=row0,rack=rack0) (2 osds) is down 2024-09-10T19:44:32.229 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph health 2024-09-10T19:44:32.229 INFO:tasks.workunit.client.0.smithi082.stderr:+ grep '1 host' 2024-09-10T19:44:32.688 INFO:tasks.workunit.client.0.smithi082.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2024-09-10T19:44:32.688 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph health 2024-09-10T19:44:32.688 INFO:tasks.workunit.client.0.smithi082.stderr:+ grep '2 osds' 2024-09-10T19:44:33.140 INFO:tasks.workunit.client.0.smithi082.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2024-09-10T19:44:33.140 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph health detail 2024-09-10T19:44:33.140 INFO:tasks.workunit.client.0.smithi082.stderr:+ grep osd.0 2024-09-10T19:44:33.658 INFO:tasks.workunit.client.0.smithi082.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2024-09-10T19:44:33.659 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph health detail 2024-09-10T19:44:33.659 INFO:tasks.workunit.client.0.smithi082.stderr:+ grep osd.1 2024-09-10T19:44:33.769 INFO:tasks.ceph.osd.0.smithi082.stderr:2024-09-10T19:44:33.766+0000 7fdeca7d2640 -1 osd.0 57 osdmap NOUP flag is set, waiting for it to clear 2024-09-10T19:44:34.178 INFO:tasks.workunit.client.0.smithi082.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2024-09-10T19:44:34.179 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph osd unset noup 2024-09-10T19:44:34.225 INFO:tasks.ceph.osd.1.smithi082.stderr:2024-09-10T19:44:34.223+0000 7fcea7721640 -1 osd.1 57 osdmap NOUP flag is set, waiting for it to clear 2024-09-10T19:44:34.609 INFO:tasks.ceph.osd.1.smithi082.stderr:2024-09-10T19:44:34.607+0000 7fce9ed2c640 -1 osd.1 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:44:34.611 INFO:tasks.ceph.osd.0.smithi082.stderr:2024-09-10T19:44:34.609+0000 7fdec5de5640 -1 osd.0 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:44:35.604 INFO:tasks.workunit.client.0.smithi082.stderr:noup is unset 2024-09-10T19:44:35.622 INFO:tasks.workunit.client.0.smithi082.stderr:+ wait_for_healthy 2024-09-10T19:44:35.622 INFO:tasks.workunit.client.0.smithi082.stderr:+ ceph health 2024-09-10T19:44:35.622 INFO:tasks.workunit.client.0.smithi082.stderr:+ grep down 2024-09-10T19:44:36.078 INFO:tasks.workunit.client.0.smithi082.stderr:+ exit 0 2024-09-10T19:44:36.078 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-10T19:44:36.079 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-10T19:44:36.143 INFO:tasks.workunit:Stopping ['rados/test_health_warnings.sh'] on client.0... 2024-09-10T19:44:36.143 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-10T19:44:36.459 DEBUG:teuthology.parallel:result is None 2024-09-10T19:44:36.460 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-10T19:44:36.484 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-10T19:44:36.484 DEBUG:teuthology.orchestra.run.smithi082:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-10T19:44:36.549 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-10T19:44:36.549 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-10T19:44:36.559 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-10T19:44:36.559 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T19:44:36.874 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:44:36.875 INFO:teuthology.orchestra.run.smithi082.stderr:dumped all 2024-09-10T19:44:36.887 INFO:teuthology.orchestra.run.smithi082.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2024-09-10T19:44:36.603401+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":276380,"kb_used_data":6048,"kb_used_omap":15,"kb_used_meta":270064,"kb_avail":1048299620,"statfs":{"total":1073741824000,"available":1073458810880,"internally_reserved":0,"allocated":6193152,"data_stored":2910034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":276545973},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.015910"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T19:44:32.588559+0000","last_change":"2024-09-10T19:44:22.571246+0000","last_active":"2024-09-10T19:44:32.588559+0000","last_peered":"2024-09-10T19:44:32.588559+0000","last_clean":"2024-09-10T19:44:32.588559+0000","last_became_active":"2024-09-10T19:44:22.570915+0000","last_became_peered":"2024-09-10T19:44:22.570915+0000","last_unstale":"2024-09-10T19:44:32.588559+0000","last_undegraded":"2024-09-10T19:44:32.588559+0000","last_fullsized":"2024-09-10T19:44:32.588559+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:52:49.080379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047593899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":90,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T19:44:32.591256+0000","last_change":"2024-09-10T19:44:22.574209+0000","last_active":"2024-09-10T19:44:32.591256+0000","last_peered":"2024-09-10T19:44:32.591256+0000","last_clean":"2024-09-10T19:44:32.591256+0000","last_became_active":"2024-09-10T19:44:22.573940+0000","last_became_peered":"2024-09-10T19:44:22.573940+0000","last_unstale":"2024-09-10T19:44:32.591256+0000","last_undegraded":"2024-09-10T19:44:32.591256+0000","last_fullsized":"2024-09-10T19:44:32.591256+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:03.057908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041823700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":94,"reported_epoch":56,"state":"active+undersized+wait","last_fresh":"2024-09-10T19:44:32.591911+0000","last_change":"2024-09-10T19:44:32.591758+0000","last_active":"2024-09-10T19:44:32.591911+0000","last_peered":"2024-09-10T19:44:32.591911+0000","last_clean":"2024-09-10T19:44:31.064077+0000","last_became_active":"2024-09-10T19:44:32.591758+0000","last_became_peered":"2024-09-10T19:44:32.591758+0000","last_unstale":"2024-09-10T19:44:32.591911+0000","last_undegraded":"2024-09-10T19:44:32.591911+0000","last_fullsized":"2024-09-10T19:44:32.591289+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:25:58.974760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001104196,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":56,"state":"active+undersized+wait","last_fresh":"2024-09-10T19:44:32.594328+0000","last_change":"2024-09-10T19:44:32.593920+0000","last_active":"2024-09-10T19:44:32.594328+0000","last_peered":"2024-09-10T19:44:32.594328+0000","last_clean":"2024-09-10T19:44:17.130803+0000","last_became_active":"2024-09-10T19:44:32.593920+0000","last_became_peered":"2024-09-10T19:44:32.593920+0000","last_unstale":"2024-09-10T19:44:32.594328+0000","last_undegraded":"2024-09-10T19:44:32.594328+0000","last_fullsized":"2024-09-10T19:44:32.591291+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:52:49.080379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042875399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"19'1","reported_seq":80,"reported_epoch":54,"state":"peering","last_fresh":"2024-09-10T19:44:32.183408+0000","last_change":"2024-09-10T19:44:32.589028+0000","last_active":"2024-09-10T19:44:32.589028+0000","last_peered":"2024-09-10T19:44:32.183408+0000","last_clean":"2024-09-10T19:44:32.183408+0000","last_became_active":"2024-09-10T19:44:22.568247+0000","last_became_peered":"2024-09-10T19:44:22.568247+0000","last_unstale":"2024-09-10T19:44:32.183408+0000","last_undegraded":"2024-09-10T19:44:32.183408+0000","last_fullsized":"2024-09-10T19:44:32.183408+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092825199999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"21'2","reported_seq":90,"reported_epoch":56,"state":"active+undersized+degraded+wait","last_fresh":"2024-09-10T19:44:32.594150+0000","last_change":"2024-09-10T19:44:32.593984+0000","last_active":"2024-09-10T19:44:32.594150+0000","last_peered":"2024-09-10T19:44:32.594150+0000","last_clean":"2024-09-10T19:44:32.183005+0000","last_became_active":"2024-09-10T19:44:32.593984+0000","last_became_peered":"2024-09-10T19:44:32.593984+0000","last_unstale":"2024-09-10T19:44:32.594150+0000","last_undegraded":"2024-09-10T19:44:32.591522+0000","last_fullsized":"2024-09-10T19:44:32.591438+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:58.497826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084709599999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":69,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T19:44:32.982504+0000","last_change":"2024-09-10T19:44:22.571580+0000","last_active":"2024-09-10T19:44:32.982504+0000","last_peered":"2024-09-10T19:44:32.982504+0000","last_clean":"2024-09-10T19:44:32.982504+0000","last_became_active":"2024-09-10T19:44:22.571306+0000","last_became_peered":"2024-09-10T19:44:22.571306+0000","last_unstale":"2024-09-10T19:44:32.982504+0000","last_undegraded":"2024-09-10T19:44:32.982504+0000","last_fullsized":"2024-09-10T19:44:32.982504+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:24.572803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048965100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":85,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T19:44:34.834880+0000","last_change":"2024-09-10T19:44:22.574811+0000","last_active":"2024-09-10T19:44:34.834880+0000","last_peered":"2024-09-10T19:44:34.834880+0000","last_clean":"2024-09-10T19:44:34.834880+0000","last_became_active":"2024-09-10T19:44:22.574503+0000","last_became_peered":"2024-09-10T19:44:22.574503+0000","last_unstale":"2024-09-10T19:44:34.834880+0000","last_undegraded":"2024-09-10T19:44:34.834880+0000","last_fullsized":"2024-09-10T19:44:34.834880+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:20:04.316213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000443783,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":12,"acting":12,"num_store_stats":9}],"osd_stats":[{"osd":5,"up_from":46,"seq":197568495641,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27652,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829948,"statfs":{"total":107374182400,"available":107345866752,"internally_reserved":0,"allocated":630784,"data_stored":298690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":39,"seq":167503724567,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27644,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829956,"statfs":{"total":107374182400,"available":107345874944,"internally_reserved":0,"allocated":630784,"data_stored":298690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":46,"seq":197568495638,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27648,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829952,"statfs":{"total":107374182400,"available":107345870848,"internally_reserved":0,"allocated":630784,"data_stored":298690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":22,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27624,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829976,"statfs":{"total":107374182400,"available":107345895424,"internally_reserved":0,"allocated":610304,"data_stored":283313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495638,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27640,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829960,"statfs":{"total":107374182400,"available":107345879040,"internally_reserved":0,"allocated":618496,"data_stored":291030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":46,"seq":197568495639,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27632,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829968,"statfs":{"total":107374182400,"available":107345887232,"internally_reserved":0,"allocated":610304,"data_stored":283313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":46,"seq":197568495638,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27628,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829972,"statfs":{"total":107374182400,"available":107345891328,"internally_reserved":0,"allocated":606208,"data_stored":283294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":48,"seq":206158430230,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27640,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829960,"statfs":{"total":107374182400,"available":107345879040,"internally_reserved":0,"allocated":618496,"data_stored":291030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":46,"seq":197568495640,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27652,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829948,"statfs":{"total":107374182400,"available":107345866752,"internally_reserved":0,"allocated":630784,"data_stored":298690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":46,"seq":197568495639,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27620,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829980,"statfs":{"total":107374182400,"available":107345899520,"internally_reserved":0,"allocated":606208,"data_stored":283294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:44:36.888 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T19:44:37.172 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:44:37.173 INFO:teuthology.orchestra.run.smithi082.stderr:dumped all 2024-09-10T19:44:37.185 INFO:teuthology.orchestra.run.smithi082.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-09-10T19:44:37.024162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":276380,"kb_used_data":6048,"kb_used_omap":15,"kb_used_meta":270064,"kb_avail":1048299620,"statfs":{"total":1073741824000,"available":1073458810880,"internally_reserved":0,"allocated":6193152,"data_stored":2910034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":276545973},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000782"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T19:44:32.588559+0000","last_change":"2024-09-10T19:44:22.571246+0000","last_active":"2024-09-10T19:44:32.588559+0000","last_peered":"2024-09-10T19:44:32.588559+0000","last_clean":"2024-09-10T19:44:32.588559+0000","last_became_active":"2024-09-10T19:44:22.570915+0000","last_became_peered":"2024-09-10T19:44:22.570915+0000","last_unstale":"2024-09-10T19:44:32.588559+0000","last_undegraded":"2024-09-10T19:44:32.588559+0000","last_fullsized":"2024-09-10T19:44:32.588559+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:52:49.080379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047593899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":90,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T19:44:32.591256+0000","last_change":"2024-09-10T19:44:22.574209+0000","last_active":"2024-09-10T19:44:32.591256+0000","last_peered":"2024-09-10T19:44:32.591256+0000","last_clean":"2024-09-10T19:44:32.591256+0000","last_became_active":"2024-09-10T19:44:22.573940+0000","last_became_peered":"2024-09-10T19:44:22.573940+0000","last_unstale":"2024-09-10T19:44:32.591256+0000","last_undegraded":"2024-09-10T19:44:32.591256+0000","last_fullsized":"2024-09-10T19:44:32.591256+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:03.057908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041823700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":94,"reported_epoch":56,"state":"active+undersized+wait","last_fresh":"2024-09-10T19:44:32.591911+0000","last_change":"2024-09-10T19:44:32.591758+0000","last_active":"2024-09-10T19:44:32.591911+0000","last_peered":"2024-09-10T19:44:32.591911+0000","last_clean":"2024-09-10T19:44:31.064077+0000","last_became_active":"2024-09-10T19:44:32.591758+0000","last_became_peered":"2024-09-10T19:44:32.591758+0000","last_unstale":"2024-09-10T19:44:32.591911+0000","last_undegraded":"2024-09-10T19:44:32.591911+0000","last_fullsized":"2024-09-10T19:44:32.591289+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:25:58.974760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001104196,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":56,"state":"active+undersized+wait","last_fresh":"2024-09-10T19:44:32.594328+0000","last_change":"2024-09-10T19:44:32.593920+0000","last_active":"2024-09-10T19:44:32.594328+0000","last_peered":"2024-09-10T19:44:32.594328+0000","last_clean":"2024-09-10T19:44:17.130803+0000","last_became_active":"2024-09-10T19:44:32.593920+0000","last_became_peered":"2024-09-10T19:44:32.593920+0000","last_unstale":"2024-09-10T19:44:32.594328+0000","last_undegraded":"2024-09-10T19:44:32.594328+0000","last_fullsized":"2024-09-10T19:44:32.591291+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:52:49.080379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042875399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"19'1","reported_seq":80,"reported_epoch":54,"state":"peering","last_fresh":"2024-09-10T19:44:32.183408+0000","last_change":"2024-09-10T19:44:32.589028+0000","last_active":"2024-09-10T19:44:32.589028+0000","last_peered":"2024-09-10T19:44:32.183408+0000","last_clean":"2024-09-10T19:44:32.183408+0000","last_became_active":"2024-09-10T19:44:22.568247+0000","last_became_peered":"2024-09-10T19:44:22.568247+0000","last_unstale":"2024-09-10T19:44:32.183408+0000","last_undegraded":"2024-09-10T19:44:32.183408+0000","last_fullsized":"2024-09-10T19:44:32.183408+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092825199999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"21'2","reported_seq":90,"reported_epoch":56,"state":"active+undersized+degraded+wait","last_fresh":"2024-09-10T19:44:32.594150+0000","last_change":"2024-09-10T19:44:32.593984+0000","last_active":"2024-09-10T19:44:32.594150+0000","last_peered":"2024-09-10T19:44:32.594150+0000","last_clean":"2024-09-10T19:44:32.183005+0000","last_became_active":"2024-09-10T19:44:32.593984+0000","last_became_peered":"2024-09-10T19:44:32.593984+0000","last_unstale":"2024-09-10T19:44:32.594150+0000","last_undegraded":"2024-09-10T19:44:32.591522+0000","last_fullsized":"2024-09-10T19:44:32.591438+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:58.497826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084709599999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":69,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T19:44:32.982504+0000","last_change":"2024-09-10T19:44:22.571580+0000","last_active":"2024-09-10T19:44:32.982504+0000","last_peered":"2024-09-10T19:44:32.982504+0000","last_clean":"2024-09-10T19:44:32.982504+0000","last_became_active":"2024-09-10T19:44:22.571306+0000","last_became_peered":"2024-09-10T19:44:22.571306+0000","last_unstale":"2024-09-10T19:44:32.982504+0000","last_undegraded":"2024-09-10T19:44:32.982504+0000","last_fullsized":"2024-09-10T19:44:32.982504+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:24.572803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048965100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":85,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T19:44:34.834880+0000","last_change":"2024-09-10T19:44:22.574811+0000","last_active":"2024-09-10T19:44:34.834880+0000","last_peered":"2024-09-10T19:44:34.834880+0000","last_clean":"2024-09-10T19:44:34.834880+0000","last_became_active":"2024-09-10T19:44:22.574503+0000","last_became_peered":"2024-09-10T19:44:22.574503+0000","last_unstale":"2024-09-10T19:44:34.834880+0000","last_undegraded":"2024-09-10T19:44:34.834880+0000","last_fullsized":"2024-09-10T19:44:34.834880+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:20:04.316213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000443783,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":12,"acting":12,"num_store_stats":9}],"osd_stats":[{"osd":5,"up_from":46,"seq":197568495641,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27652,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829948,"statfs":{"total":107374182400,"available":107345866752,"internally_reserved":0,"allocated":630784,"data_stored":298690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":39,"seq":167503724567,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27644,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829956,"statfs":{"total":107374182400,"available":107345874944,"internally_reserved":0,"allocated":630784,"data_stored":298690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":46,"seq":197568495638,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27648,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829952,"statfs":{"total":107374182400,"available":107345870848,"internally_reserved":0,"allocated":630784,"data_stored":298690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":22,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27624,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829976,"statfs":{"total":107374182400,"available":107345895424,"internally_reserved":0,"allocated":610304,"data_stored":283313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495638,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27640,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829960,"statfs":{"total":107374182400,"available":107345879040,"internally_reserved":0,"allocated":618496,"data_stored":291030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":46,"seq":197568495639,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27632,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829968,"statfs":{"total":107374182400,"available":107345887232,"internally_reserved":0,"allocated":610304,"data_stored":283313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":46,"seq":197568495638,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27628,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829972,"statfs":{"total":107374182400,"available":107345891328,"internally_reserved":0,"allocated":606208,"data_stored":283294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":48,"seq":206158430230,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27640,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829960,"statfs":{"total":107374182400,"available":107345879040,"internally_reserved":0,"allocated":618496,"data_stored":291030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":46,"seq":197568495640,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27652,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829948,"statfs":{"total":107374182400,"available":107345866752,"internally_reserved":0,"allocated":630784,"data_stored":298690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":46,"seq":197568495639,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27620,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829980,"statfs":{"total":107374182400,"available":107345899520,"internally_reserved":0,"allocated":606208,"data_stored":283294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:44:37.186 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T19:44:37.468 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:44:37.468 INFO:teuthology.orchestra.run.smithi082.stderr:dumped all 2024-09-10T19:44:37.480 INFO:teuthology.orchestra.run.smithi082.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-09-10T19:44:37.024162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":276380,"kb_used_data":6048,"kb_used_omap":15,"kb_used_meta":270064,"kb_avail":1048299620,"statfs":{"total":1073741824000,"available":1073458810880,"internally_reserved":0,"allocated":6193152,"data_stored":2910034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":276545973},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000782"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T19:44:32.588559+0000","last_change":"2024-09-10T19:44:22.571246+0000","last_active":"2024-09-10T19:44:32.588559+0000","last_peered":"2024-09-10T19:44:32.588559+0000","last_clean":"2024-09-10T19:44:32.588559+0000","last_became_active":"2024-09-10T19:44:22.570915+0000","last_became_peered":"2024-09-10T19:44:22.570915+0000","last_unstale":"2024-09-10T19:44:32.588559+0000","last_undegraded":"2024-09-10T19:44:32.588559+0000","last_fullsized":"2024-09-10T19:44:32.588559+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:52:49.080379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047593899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":90,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T19:44:32.591256+0000","last_change":"2024-09-10T19:44:22.574209+0000","last_active":"2024-09-10T19:44:32.591256+0000","last_peered":"2024-09-10T19:44:32.591256+0000","last_clean":"2024-09-10T19:44:32.591256+0000","last_became_active":"2024-09-10T19:44:22.573940+0000","last_became_peered":"2024-09-10T19:44:22.573940+0000","last_unstale":"2024-09-10T19:44:32.591256+0000","last_undegraded":"2024-09-10T19:44:32.591256+0000","last_fullsized":"2024-09-10T19:44:32.591256+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:03.057908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041823700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":94,"reported_epoch":56,"state":"active+undersized+wait","last_fresh":"2024-09-10T19:44:32.591911+0000","last_change":"2024-09-10T19:44:32.591758+0000","last_active":"2024-09-10T19:44:32.591911+0000","last_peered":"2024-09-10T19:44:32.591911+0000","last_clean":"2024-09-10T19:44:31.064077+0000","last_became_active":"2024-09-10T19:44:32.591758+0000","last_became_peered":"2024-09-10T19:44:32.591758+0000","last_unstale":"2024-09-10T19:44:32.591911+0000","last_undegraded":"2024-09-10T19:44:32.591911+0000","last_fullsized":"2024-09-10T19:44:32.591289+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:25:58.974760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001104196,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":56,"state":"active+undersized+wait","last_fresh":"2024-09-10T19:44:32.594328+0000","last_change":"2024-09-10T19:44:32.593920+0000","last_active":"2024-09-10T19:44:32.594328+0000","last_peered":"2024-09-10T19:44:32.594328+0000","last_clean":"2024-09-10T19:44:17.130803+0000","last_became_active":"2024-09-10T19:44:32.593920+0000","last_became_peered":"2024-09-10T19:44:32.593920+0000","last_unstale":"2024-09-10T19:44:32.594328+0000","last_undegraded":"2024-09-10T19:44:32.594328+0000","last_fullsized":"2024-09-10T19:44:32.591291+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:52:49.080379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042875399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"19'1","reported_seq":80,"reported_epoch":54,"state":"peering","last_fresh":"2024-09-10T19:44:32.183408+0000","last_change":"2024-09-10T19:44:32.589028+0000","last_active":"2024-09-10T19:44:32.589028+0000","last_peered":"2024-09-10T19:44:32.183408+0000","last_clean":"2024-09-10T19:44:32.183408+0000","last_became_active":"2024-09-10T19:44:22.568247+0000","last_became_peered":"2024-09-10T19:44:22.568247+0000","last_unstale":"2024-09-10T19:44:32.183408+0000","last_undegraded":"2024-09-10T19:44:32.183408+0000","last_fullsized":"2024-09-10T19:44:32.183408+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092825199999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"21'2","reported_seq":90,"reported_epoch":56,"state":"active+undersized+degraded+wait","last_fresh":"2024-09-10T19:44:32.594150+0000","last_change":"2024-09-10T19:44:32.593984+0000","last_active":"2024-09-10T19:44:32.594150+0000","last_peered":"2024-09-10T19:44:32.594150+0000","last_clean":"2024-09-10T19:44:32.183005+0000","last_became_active":"2024-09-10T19:44:32.593984+0000","last_became_peered":"2024-09-10T19:44:32.593984+0000","last_unstale":"2024-09-10T19:44:32.594150+0000","last_undegraded":"2024-09-10T19:44:32.591522+0000","last_fullsized":"2024-09-10T19:44:32.591438+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:58.497826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084709599999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":69,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T19:44:32.982504+0000","last_change":"2024-09-10T19:44:22.571580+0000","last_active":"2024-09-10T19:44:32.982504+0000","last_peered":"2024-09-10T19:44:32.982504+0000","last_clean":"2024-09-10T19:44:32.982504+0000","last_became_active":"2024-09-10T19:44:22.571306+0000","last_became_peered":"2024-09-10T19:44:22.571306+0000","last_unstale":"2024-09-10T19:44:32.982504+0000","last_undegraded":"2024-09-10T19:44:32.982504+0000","last_fullsized":"2024-09-10T19:44:32.982504+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:24.572803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048965100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":85,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T19:44:34.834880+0000","last_change":"2024-09-10T19:44:22.574811+0000","last_active":"2024-09-10T19:44:34.834880+0000","last_peered":"2024-09-10T19:44:34.834880+0000","last_clean":"2024-09-10T19:44:34.834880+0000","last_became_active":"2024-09-10T19:44:22.574503+0000","last_became_peered":"2024-09-10T19:44:22.574503+0000","last_unstale":"2024-09-10T19:44:34.834880+0000","last_undegraded":"2024-09-10T19:44:34.834880+0000","last_fullsized":"2024-09-10T19:44:34.834880+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:20:04.316213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000443783,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":12,"acting":12,"num_store_stats":9}],"osd_stats":[{"osd":5,"up_from":46,"seq":197568495641,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27652,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829948,"statfs":{"total":107374182400,"available":107345866752,"internally_reserved":0,"allocated":630784,"data_stored":298690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":39,"seq":167503724567,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27644,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829956,"statfs":{"total":107374182400,"available":107345874944,"internally_reserved":0,"allocated":630784,"data_stored":298690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":46,"seq":197568495638,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27648,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829952,"statfs":{"total":107374182400,"available":107345870848,"internally_reserved":0,"allocated":630784,"data_stored":298690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":22,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27624,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829976,"statfs":{"total":107374182400,"available":107345895424,"internally_reserved":0,"allocated":610304,"data_stored":283313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495638,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27640,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829960,"statfs":{"total":107374182400,"available":107345879040,"internally_reserved":0,"allocated":618496,"data_stored":291030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":46,"seq":197568495639,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27632,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829968,"statfs":{"total":107374182400,"available":107345887232,"internally_reserved":0,"allocated":610304,"data_stored":283313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":46,"seq":197568495638,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27628,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829972,"statfs":{"total":107374182400,"available":107345891328,"internally_reserved":0,"allocated":606208,"data_stored":283294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":48,"seq":206158430230,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27640,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829960,"statfs":{"total":107374182400,"available":107345879040,"internally_reserved":0,"allocated":618496,"data_stored":291030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":46,"seq":197568495640,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27652,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829948,"statfs":{"total":107374182400,"available":107345866752,"internally_reserved":0,"allocated":630784,"data_stored":298690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":46,"seq":197568495639,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27620,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829980,"statfs":{"total":107374182400,"available":107345899520,"internally_reserved":0,"allocated":606208,"data_stored":283294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:44:37.481 INFO:tasks.ceph.ceph_manager.ceph:PG 1.5 is not active+clean 2024-09-10T19:44:37.481 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.5', 'version': "0'0", 'reported_seq': 94, 'reported_epoch': 56, 'state': 'active+undersized+wait', 'last_fresh': '2024-09-10T19:44:32.591911+0000', 'last_change': '2024-09-10T19:44:32.591758+0000', 'last_active': '2024-09-10T19:44:32.591911+0000', 'last_peered': '2024-09-10T19:44:32.591911+0000', 'last_clean': '2024-09-10T19:44:31.064077+0000', 'last_became_active': '2024-09-10T19:44:32.591758+0000', 'last_became_peered': '2024-09-10T19:44:32.591758+0000', 'last_unstale': '2024-09-10T19:44:32.591911+0000', 'last_undegraded': '2024-09-10T19:44:32.591911+0000', 'last_fullsized': '2024-09-10T19:44:32.591289+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 47, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T19:42:55.225356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T19:42:55.225356+0000', 'last_clean_scrub_stamp': '2024-09-10T19:42:55.225356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-12T02:25:58.974760+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001104196, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [9], 'acting': [9], 'avail_no_missing': ['9'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 9, 'acting_primary': 9, 'purged_snaps': []} 2024-09-10T19:44:37.482 INFO:tasks.ceph.ceph_manager.ceph:PG 1.4 is not active+clean 2024-09-10T19:44:37.482 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.4', 'version': "0'0", 'reported_seq': 59, 'reported_epoch': 56, 'state': 'active+undersized+wait', 'last_fresh': '2024-09-10T19:44:32.594328+0000', 'last_change': '2024-09-10T19:44:32.593920+0000', 'last_active': '2024-09-10T19:44:32.594328+0000', 'last_peered': '2024-09-10T19:44:32.594328+0000', 'last_clean': '2024-09-10T19:44:17.130803+0000', 'last_became_active': '2024-09-10T19:44:32.593920+0000', 'last_became_peered': '2024-09-10T19:44:32.593920+0000', 'last_unstale': '2024-09-10T19:44:32.594328+0000', 'last_undegraded': '2024-09-10T19:44:32.594328+0000', 'last_fullsized': '2024-09-10T19:44:32.591291+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 47, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T19:42:55.225356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T19:42:55.225356+0000', 'last_clean_scrub_stamp': '2024-09-10T19:42:55.225356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-12T03:52:49.080379+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000428754, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-10T19:44:37.482 INFO:tasks.ceph.ceph_manager.ceph:PG 1.3 is not active+clean 2024-09-10T19:44:37.482 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.3', 'version': "19'1", 'reported_seq': 80, 'reported_epoch': 54, 'state': 'peering', 'last_fresh': '2024-09-10T19:44:32.183408+0000', 'last_change': '2024-09-10T19:44:32.589028+0000', 'last_active': '2024-09-10T19:44:32.589028+0000', 'last_peered': '2024-09-10T19:44:32.183408+0000', 'last_clean': '2024-09-10T19:44:32.183408+0000', 'last_became_active': '2024-09-10T19:44:22.568247+0000', 'last_became_peered': '2024-09-10T19:44:22.568247+0000', 'last_unstale': '2024-09-10T19:44:32.183408+0000', 'last_undegraded': '2024-09-10T19:44:32.183408+0000', 'last_fullsized': '2024-09-10T19:44:32.183408+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 47, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T19:42:55.225356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T19:42:55.225356+0000', 'last_clean_scrub_stamp': '2024-09-10T19:42:55.225356+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000928252, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-10T19:44:37.482 INFO:tasks.ceph.ceph_manager.ceph:PG 1.2 is not active+clean 2024-09-10T19:44:37.482 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.2', 'version': "21'2", 'reported_seq': 90, 'reported_epoch': 56, 'state': 'active+undersized+degraded+wait', 'last_fresh': '2024-09-10T19:44:32.594150+0000', 'last_change': '2024-09-10T19:44:32.593984+0000', 'last_active': '2024-09-10T19:44:32.594150+0000', 'last_peered': '2024-09-10T19:44:32.594150+0000', 'last_clean': '2024-09-10T19:44:32.183005+0000', 'last_became_active': '2024-09-10T19:44:32.593984+0000', 'last_became_peered': '2024-09-10T19:44:32.593984+0000', 'last_unstale': '2024-09-10T19:44:32.594150+0000', 'last_undegraded': '2024-09-10T19:44:32.591522+0000', 'last_fullsized': '2024-09-10T19:44:32.591438+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 47, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-10T19:42:55.225356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-10T19:42:55.225356+0000', 'last_clean_scrub_stamp': '2024-09-10T19:42:55.225356+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-11T21:22:58.497826+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000847096, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4], 'acting': [4], 'avail_no_missing': ['4'], 'object_location_counts': [{'shards': '4', 'objects': 1}], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-10T19:44:37.482 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-10T19:44:37.832 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:44:37.832 INFO:teuthology.orchestra.run.smithi082.stdout:{"fsid":"2964cb92-693e-4d63-a589-7427632fbccd","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout flag(s) set","count":5},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1/4 objects degraded (25.000%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":115,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":60,"num_osds":10,"num_up_osds":10,"osd_up_since":1725997475,"num_in_osds":10,"osd_in_since":1725997365,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":4},{"state_name":"active+undersized+wait","count":2},{"state_name":"active+undersized+degraded+wait","count":1},{"state_name":"peering","count":1}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":283013120,"bytes_avail":1073458810880,"bytes_total":1073741824000,"inactive_pgs_ratio":0.125,"degraded_objects":1,"degraded_total":4,"degraded_ratio":0.25},"fsmap":{"epoch":1,"btime":"2024-09-10T19:42:42:419805+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-10T19:43:57.017636+0000","services":{}},"progress_events":{}} 2024-09-10T19:44:37.845 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-10T19:44:37.845 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T19:44:38.126 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:44:38.126 INFO:teuthology.orchestra.run.smithi082.stderr:dumped all 2024-09-10T19:44:38.138 INFO:teuthology.orchestra.run.smithi082.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-09-10T19:44:37.024162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":276380,"kb_used_data":6048,"kb_used_omap":15,"kb_used_meta":270064,"kb_avail":1048299620,"statfs":{"total":1073741824000,"available":1073458810880,"internally_reserved":0,"allocated":6193152,"data_stored":2910034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":276545973},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000782"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T19:44:32.588559+0000","last_change":"2024-09-10T19:44:22.571246+0000","last_active":"2024-09-10T19:44:32.588559+0000","last_peered":"2024-09-10T19:44:32.588559+0000","last_clean":"2024-09-10T19:44:32.588559+0000","last_became_active":"2024-09-10T19:44:22.570915+0000","last_became_peered":"2024-09-10T19:44:22.570915+0000","last_unstale":"2024-09-10T19:44:32.588559+0000","last_undegraded":"2024-09-10T19:44:32.588559+0000","last_fullsized":"2024-09-10T19:44:32.588559+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:52:49.080379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047593899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":90,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T19:44:32.591256+0000","last_change":"2024-09-10T19:44:22.574209+0000","last_active":"2024-09-10T19:44:32.591256+0000","last_peered":"2024-09-10T19:44:32.591256+0000","last_clean":"2024-09-10T19:44:32.591256+0000","last_became_active":"2024-09-10T19:44:22.573940+0000","last_became_peered":"2024-09-10T19:44:22.573940+0000","last_unstale":"2024-09-10T19:44:32.591256+0000","last_undegraded":"2024-09-10T19:44:32.591256+0000","last_fullsized":"2024-09-10T19:44:32.591256+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:03.057908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041823700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":94,"reported_epoch":56,"state":"active+undersized+wait","last_fresh":"2024-09-10T19:44:32.591911+0000","last_change":"2024-09-10T19:44:32.591758+0000","last_active":"2024-09-10T19:44:32.591911+0000","last_peered":"2024-09-10T19:44:32.591911+0000","last_clean":"2024-09-10T19:44:31.064077+0000","last_became_active":"2024-09-10T19:44:32.591758+0000","last_became_peered":"2024-09-10T19:44:32.591758+0000","last_unstale":"2024-09-10T19:44:32.591911+0000","last_undegraded":"2024-09-10T19:44:32.591911+0000","last_fullsized":"2024-09-10T19:44:32.591289+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:25:58.974760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001104196,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":56,"state":"active+undersized+wait","last_fresh":"2024-09-10T19:44:32.594328+0000","last_change":"2024-09-10T19:44:32.593920+0000","last_active":"2024-09-10T19:44:32.594328+0000","last_peered":"2024-09-10T19:44:32.594328+0000","last_clean":"2024-09-10T19:44:17.130803+0000","last_became_active":"2024-09-10T19:44:32.593920+0000","last_became_peered":"2024-09-10T19:44:32.593920+0000","last_unstale":"2024-09-10T19:44:32.594328+0000","last_undegraded":"2024-09-10T19:44:32.594328+0000","last_fullsized":"2024-09-10T19:44:32.591291+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:52:49.080379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042875399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"19'1","reported_seq":80,"reported_epoch":54,"state":"peering","last_fresh":"2024-09-10T19:44:32.183408+0000","last_change":"2024-09-10T19:44:32.589028+0000","last_active":"2024-09-10T19:44:32.589028+0000","last_peered":"2024-09-10T19:44:32.183408+0000","last_clean":"2024-09-10T19:44:32.183408+0000","last_became_active":"2024-09-10T19:44:22.568247+0000","last_became_peered":"2024-09-10T19:44:22.568247+0000","last_unstale":"2024-09-10T19:44:32.183408+0000","last_undegraded":"2024-09-10T19:44:32.183408+0000","last_fullsized":"2024-09-10T19:44:32.183408+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092825199999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"21'2","reported_seq":90,"reported_epoch":56,"state":"active+undersized+degraded+wait","last_fresh":"2024-09-10T19:44:32.594150+0000","last_change":"2024-09-10T19:44:32.593984+0000","last_active":"2024-09-10T19:44:32.594150+0000","last_peered":"2024-09-10T19:44:32.594150+0000","last_clean":"2024-09-10T19:44:32.183005+0000","last_became_active":"2024-09-10T19:44:32.593984+0000","last_became_peered":"2024-09-10T19:44:32.593984+0000","last_unstale":"2024-09-10T19:44:32.594150+0000","last_undegraded":"2024-09-10T19:44:32.591522+0000","last_fullsized":"2024-09-10T19:44:32.591438+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:58.497826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084709599999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":69,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-10T19:44:32.982504+0000","last_change":"2024-09-10T19:44:22.571580+0000","last_active":"2024-09-10T19:44:32.982504+0000","last_peered":"2024-09-10T19:44:32.982504+0000","last_clean":"2024-09-10T19:44:32.982504+0000","last_became_active":"2024-09-10T19:44:22.571306+0000","last_became_peered":"2024-09-10T19:44:22.571306+0000","last_unstale":"2024-09-10T19:44:32.982504+0000","last_undegraded":"2024-09-10T19:44:32.982504+0000","last_fullsized":"2024-09-10T19:44:32.982504+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:24.572803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048965100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":85,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-10T19:44:34.834880+0000","last_change":"2024-09-10T19:44:22.574811+0000","last_active":"2024-09-10T19:44:34.834880+0000","last_peered":"2024-09-10T19:44:34.834880+0000","last_clean":"2024-09-10T19:44:34.834880+0000","last_became_active":"2024-09-10T19:44:22.574503+0000","last_became_peered":"2024-09-10T19:44:22.574503+0000","last_unstale":"2024-09-10T19:44:34.834880+0000","last_undegraded":"2024-09-10T19:44:34.834880+0000","last_fullsized":"2024-09-10T19:44:34.834880+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:20:04.316213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000443783,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":12,"acting":12,"num_store_stats":9}],"osd_stats":[{"osd":5,"up_from":46,"seq":197568495641,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27652,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829948,"statfs":{"total":107374182400,"available":107345866752,"internally_reserved":0,"allocated":630784,"data_stored":298690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":39,"seq":167503724567,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27644,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829956,"statfs":{"total":107374182400,"available":107345874944,"internally_reserved":0,"allocated":630784,"data_stored":298690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":46,"seq":197568495638,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27648,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829952,"statfs":{"total":107374182400,"available":107345870848,"internally_reserved":0,"allocated":630784,"data_stored":298690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":22,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27624,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829976,"statfs":{"total":107374182400,"available":107345895424,"internally_reserved":0,"allocated":610304,"data_stored":283313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495638,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27640,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829960,"statfs":{"total":107374182400,"available":107345879040,"internally_reserved":0,"allocated":618496,"data_stored":291030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":46,"seq":197568495639,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27632,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829968,"statfs":{"total":107374182400,"available":107345887232,"internally_reserved":0,"allocated":610304,"data_stored":283313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":46,"seq":197568495638,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27628,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829972,"statfs":{"total":107374182400,"available":107345891328,"internally_reserved":0,"allocated":606208,"data_stored":283294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":48,"seq":206158430230,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27640,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829960,"statfs":{"total":107374182400,"available":107345879040,"internally_reserved":0,"allocated":618496,"data_stored":291030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":46,"seq":197568495640,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27652,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829948,"statfs":{"total":107374182400,"available":107345866752,"internally_reserved":0,"allocated":630784,"data_stored":298690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":46,"seq":197568495639,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27620,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829980,"statfs":{"total":107374182400,"available":107345899520,"internally_reserved":0,"allocated":606208,"data_stored":283294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:44:41.140 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T19:44:41.440 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:44:41.440 INFO:teuthology.orchestra.run.smithi082.stderr:dumped all 2024-09-10T19:44:41.456 INFO:teuthology.orchestra.run.smithi082.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-09-10T19:44:41.024952+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":276964,"kb_used_data":6648,"kb_used_omap":15,"kb_used_meta":270064,"kb_avail":1048299036,"statfs":{"total":1073741824000,"available":1073458212864,"internally_reserved":0,"allocated":6807552,"data_stored":3302178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":276545973},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.420630"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:36.606496+0000","last_change":"2024-09-10T19:44:22.571246+0000","last_active":"2024-09-10T19:44:36.606496+0000","last_peered":"2024-09-10T19:44:36.606496+0000","last_clean":"2024-09-10T19:44:36.606496+0000","last_became_active":"2024-09-10T19:44:22.570915+0000","last_became_peered":"2024-09-10T19:44:22.570915+0000","last_unstale":"2024-09-10T19:44:36.606496+0000","last_undegraded":"2024-09-10T19:44:36.606496+0000","last_fullsized":"2024-09-10T19:44:36.606496+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:52:49.080379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047593899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":96,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:37.083435+0000","last_change":"2024-09-10T19:44:22.574209+0000","last_active":"2024-09-10T19:44:37.083435+0000","last_peered":"2024-09-10T19:44:37.083435+0000","last_clean":"2024-09-10T19:44:37.083435+0000","last_became_active":"2024-09-10T19:44:22.573940+0000","last_became_peered":"2024-09-10T19:44:22.573940+0000","last_unstale":"2024-09-10T19:44:37.083435+0000","last_undegraded":"2024-09-10T19:44:37.083435+0000","last_fullsized":"2024-09-10T19:44:37.083435+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:03.057908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041823700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":108,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:36.722862+0000","last_change":"2024-09-10T19:44:36.722771+0000","last_active":"2024-09-10T19:44:36.722862+0000","last_peered":"2024-09-10T19:44:36.722862+0000","last_clean":"2024-09-10T19:44:36.722862+0000","last_became_active":"2024-09-10T19:44:36.722538+0000","last_became_peered":"2024-09-10T19:44:36.722538+0000","last_unstale":"2024-09-10T19:44:36.722862+0000","last_undegraded":"2024-09-10T19:44:36.722862+0000","last_fullsized":"2024-09-10T19:44:36.722862+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:15:30.176184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001104196,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":80,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:37.084888+0000","last_change":"2024-09-10T19:44:37.084786+0000","last_active":"2024-09-10T19:44:37.084888+0000","last_peered":"2024-09-10T19:44:37.084888+0000","last_clean":"2024-09-10T19:44:37.084888+0000","last_became_active":"2024-09-10T19:44:37.084491+0000","last_became_peered":"2024-09-10T19:44:37.084491+0000","last_unstale":"2024-09-10T19:44:37.084888+0000","last_undegraded":"2024-09-10T19:44:37.084888+0000","last_fullsized":"2024-09-10T19:44:37.084888+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:48:41.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042875399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"19'1","reported_seq":98,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:36.612219+0000","last_change":"2024-09-10T19:44:36.612105+0000","last_active":"2024-09-10T19:44:36.612219+0000","last_peered":"2024-09-10T19:44:36.612219+0000","last_clean":"2024-09-10T19:44:36.612219+0000","last_became_active":"2024-09-10T19:44:36.611774+0000","last_became_peered":"2024-09-10T19:44:36.611774+0000","last_unstale":"2024-09-10T19:44:36.612219+0000","last_undegraded":"2024-09-10T19:44:36.612219+0000","last_fullsized":"2024-09-10T19:44:36.612219+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:17:34.212417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092825199999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"21'2","reported_seq":104,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:36.612303+0000","last_change":"2024-09-10T19:44:36.612222+0000","last_active":"2024-09-10T19:44:36.612303+0000","last_peered":"2024-09-10T19:44:36.612303+0000","last_clean":"2024-09-10T19:44:36.612303+0000","last_became_active":"2024-09-10T19:44:36.611940+0000","last_became_peered":"2024-09-10T19:44:36.611940+0000","last_unstale":"2024-09-10T19:44:36.612303+0000","last_undegraded":"2024-09-10T19:44:36.612303+0000","last_fullsized":"2024-09-10T19:44:36.612303+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:24:47.709541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084709599999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":74,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:36.609226+0000","last_change":"2024-09-10T19:44:22.571580+0000","last_active":"2024-09-10T19:44:36.609226+0000","last_peered":"2024-09-10T19:44:36.609226+0000","last_clean":"2024-09-10T19:44:36.609226+0000","last_became_active":"2024-09-10T19:44:22.571306+0000","last_became_peered":"2024-09-10T19:44:22.571306+0000","last_unstale":"2024-09-10T19:44:36.609226+0000","last_undegraded":"2024-09-10T19:44:36.609226+0000","last_fullsized":"2024-09-10T19:44:36.609226+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:24.572803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048965100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":88,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:37.083631+0000","last_change":"2024-09-10T19:44:22.574811+0000","last_active":"2024-09-10T19:44:37.083631+0000","last_peered":"2024-09-10T19:44:37.083631+0000","last_clean":"2024-09-10T19:44:37.083631+0000","last_became_active":"2024-09-10T19:44:22.574503+0000","last_became_peered":"2024-09-10T19:44:22.574503+0000","last_unstale":"2024-09-10T19:44:37.083631+0000","last_undegraded":"2024-09-10T19:44:37.083631+0000","last_fullsized":"2024-09-10T19:44:37.083631+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:20:04.316213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000443783,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":5,"up_from":46,"seq":197568495642,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27700,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829900,"statfs":{"total":107374182400,"available":107345817600,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":39,"seq":167503724568,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27696,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829904,"statfs":{"total":107374182400,"available":107345821696,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":46,"seq":197568495639,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27692,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":59,"seq":253403070487,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27704,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829896,"statfs":{"total":107374182400,"available":107345813504,"internally_reserved":0,"allocated":684032,"data_stored":330233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495639,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27696,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829904,"statfs":{"total":107374182400,"available":107345821696,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":46,"seq":197568495640,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27700,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829900,"statfs":{"total":107374182400,"available":107345817600,"internally_reserved":0,"allocated":684032,"data_stored":330233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":46,"seq":197568495639,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27696,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829904,"statfs":{"total":107374182400,"available":107345821696,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":48,"seq":206158430231,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27692,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":46,"seq":197568495641,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27692,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":59,"seq":253403070488,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27696,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829904,"statfs":{"total":107374182400,"available":107345821696,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:44:41.457 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-10T19:44:41.458 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T19:44:41.738 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:44:41.739 INFO:teuthology.orchestra.run.smithi082.stderr:dumped all 2024-09-10T19:44:41.751 INFO:teuthology.orchestra.run.smithi082.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-09-10T19:44:41.024952+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":276964,"kb_used_data":6648,"kb_used_omap":15,"kb_used_meta":270064,"kb_avail":1048299036,"statfs":{"total":1073741824000,"available":1073458212864,"internally_reserved":0,"allocated":6807552,"data_stored":3302178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":276545973},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.420630"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:36.606496+0000","last_change":"2024-09-10T19:44:22.571246+0000","last_active":"2024-09-10T19:44:36.606496+0000","last_peered":"2024-09-10T19:44:36.606496+0000","last_clean":"2024-09-10T19:44:36.606496+0000","last_became_active":"2024-09-10T19:44:22.570915+0000","last_became_peered":"2024-09-10T19:44:22.570915+0000","last_unstale":"2024-09-10T19:44:36.606496+0000","last_undegraded":"2024-09-10T19:44:36.606496+0000","last_fullsized":"2024-09-10T19:44:36.606496+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:52:49.080379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047593899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":96,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:37.083435+0000","last_change":"2024-09-10T19:44:22.574209+0000","last_active":"2024-09-10T19:44:37.083435+0000","last_peered":"2024-09-10T19:44:37.083435+0000","last_clean":"2024-09-10T19:44:37.083435+0000","last_became_active":"2024-09-10T19:44:22.573940+0000","last_became_peered":"2024-09-10T19:44:22.573940+0000","last_unstale":"2024-09-10T19:44:37.083435+0000","last_undegraded":"2024-09-10T19:44:37.083435+0000","last_fullsized":"2024-09-10T19:44:37.083435+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:03.057908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041823700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":108,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:36.722862+0000","last_change":"2024-09-10T19:44:36.722771+0000","last_active":"2024-09-10T19:44:36.722862+0000","last_peered":"2024-09-10T19:44:36.722862+0000","last_clean":"2024-09-10T19:44:36.722862+0000","last_became_active":"2024-09-10T19:44:36.722538+0000","last_became_peered":"2024-09-10T19:44:36.722538+0000","last_unstale":"2024-09-10T19:44:36.722862+0000","last_undegraded":"2024-09-10T19:44:36.722862+0000","last_fullsized":"2024-09-10T19:44:36.722862+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:15:30.176184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001104196,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":80,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:37.084888+0000","last_change":"2024-09-10T19:44:37.084786+0000","last_active":"2024-09-10T19:44:37.084888+0000","last_peered":"2024-09-10T19:44:37.084888+0000","last_clean":"2024-09-10T19:44:37.084888+0000","last_became_active":"2024-09-10T19:44:37.084491+0000","last_became_peered":"2024-09-10T19:44:37.084491+0000","last_unstale":"2024-09-10T19:44:37.084888+0000","last_undegraded":"2024-09-10T19:44:37.084888+0000","last_fullsized":"2024-09-10T19:44:37.084888+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:48:41.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042875399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"19'1","reported_seq":98,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:36.612219+0000","last_change":"2024-09-10T19:44:36.612105+0000","last_active":"2024-09-10T19:44:36.612219+0000","last_peered":"2024-09-10T19:44:36.612219+0000","last_clean":"2024-09-10T19:44:36.612219+0000","last_became_active":"2024-09-10T19:44:36.611774+0000","last_became_peered":"2024-09-10T19:44:36.611774+0000","last_unstale":"2024-09-10T19:44:36.612219+0000","last_undegraded":"2024-09-10T19:44:36.612219+0000","last_fullsized":"2024-09-10T19:44:36.612219+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:17:34.212417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092825199999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"21'2","reported_seq":104,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:36.612303+0000","last_change":"2024-09-10T19:44:36.612222+0000","last_active":"2024-09-10T19:44:36.612303+0000","last_peered":"2024-09-10T19:44:36.612303+0000","last_clean":"2024-09-10T19:44:36.612303+0000","last_became_active":"2024-09-10T19:44:36.611940+0000","last_became_peered":"2024-09-10T19:44:36.611940+0000","last_unstale":"2024-09-10T19:44:36.612303+0000","last_undegraded":"2024-09-10T19:44:36.612303+0000","last_fullsized":"2024-09-10T19:44:36.612303+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:24:47.709541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084709599999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":74,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:36.609226+0000","last_change":"2024-09-10T19:44:22.571580+0000","last_active":"2024-09-10T19:44:36.609226+0000","last_peered":"2024-09-10T19:44:36.609226+0000","last_clean":"2024-09-10T19:44:36.609226+0000","last_became_active":"2024-09-10T19:44:22.571306+0000","last_became_peered":"2024-09-10T19:44:22.571306+0000","last_unstale":"2024-09-10T19:44:36.609226+0000","last_undegraded":"2024-09-10T19:44:36.609226+0000","last_fullsized":"2024-09-10T19:44:36.609226+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:24.572803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048965100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":88,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:37.083631+0000","last_change":"2024-09-10T19:44:22.574811+0000","last_active":"2024-09-10T19:44:37.083631+0000","last_peered":"2024-09-10T19:44:37.083631+0000","last_clean":"2024-09-10T19:44:37.083631+0000","last_became_active":"2024-09-10T19:44:22.574503+0000","last_became_peered":"2024-09-10T19:44:22.574503+0000","last_unstale":"2024-09-10T19:44:37.083631+0000","last_undegraded":"2024-09-10T19:44:37.083631+0000","last_fullsized":"2024-09-10T19:44:37.083631+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:20:04.316213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000443783,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":5,"up_from":46,"seq":197568495642,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27700,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829900,"statfs":{"total":107374182400,"available":107345817600,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":39,"seq":167503724568,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27696,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829904,"statfs":{"total":107374182400,"available":107345821696,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":46,"seq":197568495639,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27692,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":59,"seq":253403070487,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27704,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829896,"statfs":{"total":107374182400,"available":107345813504,"internally_reserved":0,"allocated":684032,"data_stored":330233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495639,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27696,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829904,"statfs":{"total":107374182400,"available":107345821696,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":46,"seq":197568495640,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27700,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829900,"statfs":{"total":107374182400,"available":107345817600,"internally_reserved":0,"allocated":684032,"data_stored":330233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":46,"seq":197568495639,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27696,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829904,"statfs":{"total":107374182400,"available":107345821696,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":48,"seq":206158430231,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27692,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":46,"seq":197568495641,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27692,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":59,"seq":253403070488,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27696,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829904,"statfs":{"total":107374182400,"available":107345821696,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:44:41.752 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T19:44:42.035 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:44:42.035 INFO:teuthology.orchestra.run.smithi082.stdout:{"epoch":60,"fsid":"2964cb92-693e-4d63-a589-7427632fbccd","created":"2024-09-10T19:42:42.419999+0000","modified":"2024-09-10T19:44:36.602378+0000","last_up_change":"2024-09-10T19:44:35.601439+0000","last_in_change":"2024-09-10T19:42:45.287001+0000","flags":"noout,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5800064,"flags_set":["noout","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":11,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-10T19:42:54.375436+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":21,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":0.80000001192092896,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1.0000001192092896,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad42803d-481b-49b9-9204-f2239ad7df2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":58,"up_from":59,"up_thru":36,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1663731654},{"type":"v1","addr":"172.21.15.82:6801","nonce":1663731654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6868","nonce":1666731654},{"type":"v1","addr":"172.21.15.82:6869","nonce":1666731654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1663731654},{"type":"v1","addr":"172.21.15.82:6807","nonce":1663731654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1663731654},{"type":"v1","addr":"172.21.15.82:6805","nonce":1663731654}]},"public_addr":"172.21.15.82:6801/1663731654","cluster_addr":"172.21.15.82:6869/1666731654","heartbeat_back_addr":"172.21.15.82:6807/1663731654","heartbeat_front_addr":"172.21.15.82:6805/1663731654","state":["exists","up"]},{"osd":1,"uuid":"c4fb1039-47d0-4f84-92ba-6d15d78a152d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":17,"last_clean_end":58,"up_from":59,"up_thru":59,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6842","nonce":3441180664},{"type":"v1","addr":"172.21.15.82:6843","nonce":3441180664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6860","nonce":3444180664},{"type":"v1","addr":"172.21.15.82:6861","nonce":3444180664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6848","nonce":3441180664},{"type":"v1","addr":"172.21.15.82:6849","nonce":3441180664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6846","nonce":3441180664},{"type":"v1","addr":"172.21.15.82:6847","nonce":3441180664}]},"public_addr":"172.21.15.82:6843/3441180664","cluster_addr":"172.21.15.82:6861/3444180664","heartbeat_back_addr":"172.21.15.82:6849/3441180664","heartbeat_front_addr":"172.21.15.82:6847/3441180664","state":["exists","up"]},{"osd":2,"uuid":"79860c95-6c4e-4b14-bbbd-0655ed2aea42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":16,"last_clean_end":47,"up_from":48,"up_thru":0,"down_at":42,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6850","nonce":3980847884},{"type":"v1","addr":"172.21.15.82:6851","nonce":3980847884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6876","nonce":3981847884},{"type":"v1","addr":"172.21.15.82:6877","nonce":3981847884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6856","nonce":3980847884},{"type":"v1","addr":"172.21.15.82:6857","nonce":3980847884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6854","nonce":3980847884},{"type":"v1","addr":"172.21.15.82:6855","nonce":3980847884}]},"public_addr":"172.21.15.82:6851/3980847884","cluster_addr":"172.21.15.82:6877/3981847884","heartbeat_back_addr":"172.21.15.82:6857/3980847884","heartbeat_front_addr":"172.21.15.82:6855/3980847884","state":["exists","up"]},{"osd":3,"uuid":"7838eabe-f0fd-422d-91dc-2ffee0d0a458","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":16,"last_clean_end":45,"up_from":46,"up_thru":59,"down_at":42,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":3682993015},{"type":"v1","addr":"172.21.15.82:6817","nonce":3682993015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":3683993015},{"type":"v1","addr":"172.21.15.82:6835","nonce":3683993015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3682993015},{"type":"v1","addr":"172.21.15.82:6826","nonce":3682993015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":3682993015},{"type":"v1","addr":"172.21.15.82:6822","nonce":3682993015}]},"public_addr":"172.21.15.82:6817/3682993015","cluster_addr":"172.21.15.82:6835/3683993015","heartbeat_back_addr":"172.21.15.82:6826/3682993015","heartbeat_front_addr":"172.21.15.82:6822/3682993015","state":["exists","up"]},{"osd":4,"uuid":"bac446ed-c130-445f-b889-9180350925e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":16,"last_clean_end":45,"up_from":46,"up_thru":59,"down_at":42,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":107394753},{"type":"v1","addr":"172.21.15.82:6832","nonce":107394753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6882","nonce":108394753},{"type":"v1","addr":"172.21.15.82:6883","nonce":108394753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":107394753},{"type":"v1","addr":"172.21.15.82:6839","nonce":107394753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":107394753},{"type":"v1","addr":"172.21.15.82:6837","nonce":107394753}]},"public_addr":"172.21.15.82:6832/107394753","cluster_addr":"172.21.15.82:6883/108394753","heartbeat_back_addr":"172.21.15.82:6839/107394753","heartbeat_front_addr":"172.21.15.82:6837/107394753","state":["exists","up"]},{"osd":5,"uuid":"c39e3a52-e1a0-4471-90fd-822b4f05199a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":17,"last_clean_end":45,"up_from":46,"up_thru":46,"down_at":42,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6821","nonce":3810102128},{"type":"v1","addr":"172.21.15.82:6823","nonce":3810102128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6844","nonce":3811102128},{"type":"v1","addr":"172.21.15.82:6845","nonce":3811102128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6831","nonce":3810102128},{"type":"v1","addr":"172.21.15.82:6833","nonce":3810102128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3810102128},{"type":"v1","addr":"172.21.15.82:6829","nonce":3810102128}]},"public_addr":"172.21.15.82:6823/3810102128","cluster_addr":"172.21.15.82:6845/3811102128","heartbeat_back_addr":"172.21.15.82:6833/3810102128","heartbeat_front_addr":"172.21.15.82:6829/3810102128","state":["exists","up"]},{"osd":6,"uuid":"4bf59a66-0ff9-4245-8405-d9fe6216075f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":16,"last_clean_end":45,"up_from":46,"up_thru":55,"down_at":42,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6866","nonce":454186391},{"type":"v1","addr":"172.21.15.82:6867","nonce":454186391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":455186391},{"type":"v1","addr":"172.21.15.82:6803","nonce":455186391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6872","nonce":454186391},{"type":"v1","addr":"172.21.15.82:6873","nonce":454186391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6870","nonce":454186391},{"type":"v1","addr":"172.21.15.82:6871","nonce":454186391}]},"public_addr":"172.21.15.82:6867/454186391","cluster_addr":"172.21.15.82:6803/455186391","heartbeat_back_addr":"172.21.15.82:6873/454186391","heartbeat_front_addr":"172.21.15.82:6871/454186391","state":["exists","up"]},{"osd":7,"uuid":"ad29323b-3d4d-4e3e-90cf-a2d57c95bce8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":17,"last_clean_end":45,"up_from":46,"up_thru":46,"down_at":42,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6874","nonce":192219834},{"type":"v1","addr":"172.21.15.82:6875","nonce":192219834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":193219834},{"type":"v1","addr":"172.21.15.82:6819","nonce":193219834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6880","nonce":192219834},{"type":"v1","addr":"172.21.15.82:6881","nonce":192219834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6878","nonce":192219834},{"type":"v1","addr":"172.21.15.82:6879","nonce":192219834}]},"public_addr":"172.21.15.82:6875/192219834","cluster_addr":"172.21.15.82:6819/193219834","heartbeat_back_addr":"172.21.15.82:6881/192219834","heartbeat_front_addr":"172.21.15.82:6879/192219834","state":["exists","up"]},{"osd":8,"uuid":"1f777d60-2f41-4f6b-a348-014322d1509e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":45,"up_from":46,"up_thru":18,"down_at":42,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":691756156},{"type":"v1","addr":"172.21.15.82:6809","nonce":691756156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6852","nonce":693756156},{"type":"v1","addr":"172.21.15.82:6853","nonce":693756156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":691756156},{"type":"v1","addr":"172.21.15.82:6815","nonce":691756156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":691756156},{"type":"v1","addr":"172.21.15.82:6813","nonce":691756156}]},"public_addr":"172.21.15.82:6809/691756156","cluster_addr":"172.21.15.82:6853/693756156","heartbeat_back_addr":"172.21.15.82:6815/691756156","heartbeat_front_addr":"172.21.15.82:6813/691756156","state":["exists","up"]},{"osd":9,"uuid":"5345f4dc-dd23-4e0a-8520-e9c675030b43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":17,"last_clean_end":38,"up_from":39,"up_thru":59,"down_at":36,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6858","nonce":2504847011},{"type":"v1","addr":"172.21.15.82:6859","nonce":2504847011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2505847011},{"type":"v1","addr":"172.21.15.82:6811","nonce":2505847011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6864","nonce":2504847011},{"type":"v1","addr":"172.21.15.82:6865","nonce":2504847011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6862","nonce":2504847011},{"type":"v1","addr":"172.21.15.82:6863","nonce":2504847011}]},"public_addr":"172.21.15.82:6859/2504847011","cluster_addr":"172.21.15.82:6811/2505847011","heartbeat_back_addr":"172.21.15.82:6865/2504847011","heartbeat_front_addr":"172.21.15.82:6863/2504847011","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2024-09-10T19:44:31.584262+0000","laggy_probability":0.65700000524520874,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:50.873734+0000","dead_epoch":56},{"osd":1,"down_stamp":"2024-09-10T19:44:31.584262+0000","laggy_probability":0.65700000524520874,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:51.362765+0000","dead_epoch":57},{"osd":2,"down_stamp":"2024-09-10T19:44:17.531399+0000","laggy_probability":0.30000001192092896,"laggy_interval":1,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:51.637626+0000","dead_epoch":45},{"osd":3,"down_stamp":"2024-09-10T19:44:17.531399+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:51.138513+0000","dead_epoch":44},{"osd":4,"down_stamp":"2024-09-10T19:44:17.531399+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:51.176514+0000","dead_epoch":43},{"osd":5,"down_stamp":"2024-09-10T19:44:17.531399+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:50.914217+0000","dead_epoch":43},{"osd":6,"down_stamp":"2024-09-10T19:44:17.531399+0000","laggy_probability":0.30000001192092896,"laggy_interval":1,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:51.068017+0000","dead_epoch":45},{"osd":7,"down_stamp":"2024-09-10T19:44:17.531399+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:51.625988+0000","dead_epoch":44},{"osd":8,"down_stamp":"2024-09-10T19:44:17.531399+0000","laggy_probability":0.50999999046325684,"laggy_interval":1,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:50.655627+0000","dead_epoch":45},{"osd":9,"down_stamp":"2024-09-10T19:44:09.504777+0000","laggy_probability":0.30000001192092896,"laggy_interval":1,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:42:51.405482+0000","dead_epoch":38}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T19:44:43.050 INFO:tasks.ceph:Scrubbing osd.0 2024-09-10T19:44:43.050 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-10T19:44:43.185 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-10T19:44:43.185 INFO:teuthology.orchestra.run.smithi082.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T19:44:43.185 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-10T19:44:43.195 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-10T19:44:43.480 INFO:teuthology.orchestra.run.smithi082.stderr:instructed osd(s) 0 to deep-scrub 2024-09-10T19:44:43.493 INFO:tasks.ceph:Scrubbing osd.1 2024-09-10T19:44:43.493 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-10T19:44:43.614 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-10T19:44:43.614 INFO:teuthology.orchestra.run.smithi082.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T19:44:43.614 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-10T19:44:43.625 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-10T19:44:43.908 INFO:teuthology.orchestra.run.smithi082.stderr:instructed osd(s) 1 to deep-scrub 2024-09-10T19:44:43.921 INFO:tasks.ceph:Scrubbing osd.2 2024-09-10T19:44:43.921 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-10T19:44:44.042 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-10T19:44:44.042 INFO:teuthology.orchestra.run.smithi082.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T19:44:44.042 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-10T19:44:44.052 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-10T19:44:44.334 INFO:teuthology.orchestra.run.smithi082.stderr:instructed osd(s) 2 to deep-scrub 2024-09-10T19:44:44.347 INFO:tasks.ceph:Scrubbing osd.3 2024-09-10T19:44:44.347 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-10T19:44:44.469 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-10T19:44:44.469 INFO:teuthology.orchestra.run.smithi082.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T19:44:44.469 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-10T19:44:44.479 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-10T19:44:44.763 INFO:teuthology.orchestra.run.smithi082.stderr:instructed osd(s) 3 to deep-scrub 2024-09-10T19:44:44.776 INFO:tasks.ceph:Scrubbing osd.4 2024-09-10T19:44:44.777 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-10T19:44:44.898 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-10T19:44:44.898 INFO:teuthology.orchestra.run.smithi082.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T19:44:44.898 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-10T19:44:44.908 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-10T19:44:45.195 INFO:teuthology.orchestra.run.smithi082.stderr:instructed osd(s) 4 to deep-scrub 2024-09-10T19:44:45.208 INFO:tasks.ceph:Scrubbing osd.5 2024-09-10T19:44:45.208 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-10T19:44:45.330 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-10T19:44:45.330 INFO:teuthology.orchestra.run.smithi082.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T19:44:45.330 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-10T19:44:45.340 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-10T19:44:45.622 INFO:teuthology.orchestra.run.smithi082.stderr:instructed osd(s) 5 to deep-scrub 2024-09-10T19:44:45.635 INFO:tasks.ceph:Scrubbing osd.6 2024-09-10T19:44:45.636 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-10T19:44:45.758 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-10T19:44:45.758 INFO:teuthology.orchestra.run.smithi082.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T19:44:45.758 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-10T19:44:45.768 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-10T19:44:46.053 INFO:teuthology.orchestra.run.smithi082.stderr:instructed osd(s) 6 to deep-scrub 2024-09-10T19:44:46.066 INFO:tasks.ceph:Scrubbing osd.7 2024-09-10T19:44:46.066 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-10T19:44:46.188 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-10T19:44:46.188 INFO:teuthology.orchestra.run.smithi082.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T19:44:46.188 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-10T19:44:46.198 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-10T19:44:46.482 INFO:teuthology.orchestra.run.smithi082.stderr:instructed osd(s) 7 to deep-scrub 2024-09-10T19:44:46.495 INFO:tasks.ceph:Scrubbing osd.8 2024-09-10T19:44:46.495 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 config set osd_debug_deep_scrub_sleep 0 2024-09-10T19:44:46.617 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-10T19:44:46.617 INFO:teuthology.orchestra.run.smithi082.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T19:44:46.617 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-10T19:44:46.628 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 8 2024-09-10T19:44:46.910 INFO:teuthology.orchestra.run.smithi082.stderr:instructed osd(s) 8 to deep-scrub 2024-09-10T19:44:46.923 INFO:tasks.ceph:Scrubbing osd.9 2024-09-10T19:44:46.923 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 config set osd_debug_deep_scrub_sleep 0 2024-09-10T19:44:47.046 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-10T19:44:47.046 INFO:teuthology.orchestra.run.smithi082.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T19:44:47.047 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-10T19:44:47.056 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 9 2024-09-10T19:44:47.342 INFO:teuthology.orchestra.run.smithi082.stderr:instructed osd(s) 9 to deep-scrub 2024-09-10T19:44:47.355 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T19:44:47.635 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:44:47.635 INFO:teuthology.orchestra.run.smithi082.stderr:dumped all 2024-09-10T19:44:47.647 INFO:teuthology.orchestra.run.smithi082.stdout:{"pg_ready":true,"pg_map":{"version":108,"stamp":"2024-09-10T19:44:47.026179+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":276964,"kb_used_data":6648,"kb_used_omap":15,"kb_used_meta":270064,"kb_avail":1048299036,"statfs":{"total":1073741824000,"available":1073458212864,"internally_reserved":0,"allocated":6807552,"data_stored":3302178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":276545973},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.422777"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:36.606496+0000","last_change":"2024-09-10T19:44:22.571246+0000","last_active":"2024-09-10T19:44:36.606496+0000","last_peered":"2024-09-10T19:44:36.606496+0000","last_clean":"2024-09-10T19:44:36.606496+0000","last_became_active":"2024-09-10T19:44:22.570915+0000","last_became_peered":"2024-09-10T19:44:22.570915+0000","last_unstale":"2024-09-10T19:44:36.606496+0000","last_undegraded":"2024-09-10T19:44:36.606496+0000","last_fullsized":"2024-09-10T19:44:36.606496+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:52:49.080379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047593899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":96,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:37.083435+0000","last_change":"2024-09-10T19:44:22.574209+0000","last_active":"2024-09-10T19:44:37.083435+0000","last_peered":"2024-09-10T19:44:37.083435+0000","last_clean":"2024-09-10T19:44:37.083435+0000","last_became_active":"2024-09-10T19:44:22.573940+0000","last_became_peered":"2024-09-10T19:44:22.573940+0000","last_unstale":"2024-09-10T19:44:37.083435+0000","last_undegraded":"2024-09-10T19:44:37.083435+0000","last_fullsized":"2024-09-10T19:44:37.083435+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:03.057908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041823700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":108,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:36.722862+0000","last_change":"2024-09-10T19:44:36.722771+0000","last_active":"2024-09-10T19:44:36.722862+0000","last_peered":"2024-09-10T19:44:36.722862+0000","last_clean":"2024-09-10T19:44:36.722862+0000","last_became_active":"2024-09-10T19:44:36.722538+0000","last_became_peered":"2024-09-10T19:44:36.722538+0000","last_unstale":"2024-09-10T19:44:36.722862+0000","last_undegraded":"2024-09-10T19:44:36.722862+0000","last_fullsized":"2024-09-10T19:44:36.722862+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:15:30.176184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001104196,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":88,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:43.921844+0000","last_change":"2024-09-10T19:44:43.921844+0000","last_active":"2024-09-10T19:44:43.921844+0000","last_peered":"2024-09-10T19:44:43.921844+0000","last_clean":"2024-09-10T19:44:43.921844+0000","last_became_active":"2024-09-10T19:44:37.084491+0000","last_became_peered":"2024-09-10T19:44:37.084491+0000","last_unstale":"2024-09-10T19:44:43.921844+0000","last_undegraded":"2024-09-10T19:44:43.921844+0000","last_fullsized":"2024-09-10T19:44:43.921844+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:44:43.921783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:44:43.921783+0000","last_clean_scrub_stamp":"2024-09-10T19:44:43.921783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:27:47.671188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042875399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"19'1","reported_seq":98,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:36.612219+0000","last_change":"2024-09-10T19:44:36.612105+0000","last_active":"2024-09-10T19:44:36.612219+0000","last_peered":"2024-09-10T19:44:36.612219+0000","last_clean":"2024-09-10T19:44:36.612219+0000","last_became_active":"2024-09-10T19:44:36.611774+0000","last_became_peered":"2024-09-10T19:44:36.611774+0000","last_unstale":"2024-09-10T19:44:36.612219+0000","last_undegraded":"2024-09-10T19:44:36.612219+0000","last_fullsized":"2024-09-10T19:44:36.612219+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:17:34.212417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092825199999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"21'2","reported_seq":104,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:36.612303+0000","last_change":"2024-09-10T19:44:36.612222+0000","last_active":"2024-09-10T19:44:36.612303+0000","last_peered":"2024-09-10T19:44:36.612303+0000","last_clean":"2024-09-10T19:44:36.612303+0000","last_became_active":"2024-09-10T19:44:36.611940+0000","last_became_peered":"2024-09-10T19:44:36.611940+0000","last_unstale":"2024-09-10T19:44:36.612303+0000","last_undegraded":"2024-09-10T19:44:36.612303+0000","last_fullsized":"2024-09-10T19:44:36.612303+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:24:47.709541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084709599999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":74,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:36.609226+0000","last_change":"2024-09-10T19:44:22.571580+0000","last_active":"2024-09-10T19:44:36.609226+0000","last_peered":"2024-09-10T19:44:36.609226+0000","last_clean":"2024-09-10T19:44:36.609226+0000","last_became_active":"2024-09-10T19:44:22.571306+0000","last_became_peered":"2024-09-10T19:44:22.571306+0000","last_unstale":"2024-09-10T19:44:36.609226+0000","last_undegraded":"2024-09-10T19:44:36.609226+0000","last_fullsized":"2024-09-10T19:44:36.609226+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:24.572803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048965100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":88,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:37.083631+0000","last_change":"2024-09-10T19:44:22.574811+0000","last_active":"2024-09-10T19:44:37.083631+0000","last_peered":"2024-09-10T19:44:37.083631+0000","last_clean":"2024-09-10T19:44:37.083631+0000","last_became_active":"2024-09-10T19:44:22.574503+0000","last_became_peered":"2024-09-10T19:44:22.574503+0000","last_unstale":"2024-09-10T19:44:37.083631+0000","last_undegraded":"2024-09-10T19:44:37.083631+0000","last_fullsized":"2024-09-10T19:44:37.083631+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:42:55.225356+0000","last_clean_scrub_stamp":"2024-09-10T19:42:55.225356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:20:04.316213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000443783,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":5,"up_from":46,"seq":197568495643,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27700,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829900,"statfs":{"total":107374182400,"available":107345817600,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":39,"seq":167503724569,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27696,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829904,"statfs":{"total":107374182400,"available":107345821696,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":46,"seq":197568495640,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27692,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":59,"seq":253403070488,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27704,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829896,"statfs":{"total":107374182400,"available":107345813504,"internally_reserved":0,"allocated":684032,"data_stored":330233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495640,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27696,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829904,"statfs":{"total":107374182400,"available":107345821696,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":46,"seq":197568495641,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27700,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829900,"statfs":{"total":107374182400,"available":107345817600,"internally_reserved":0,"allocated":684032,"data_stored":330233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":46,"seq":197568495640,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27696,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829904,"statfs":{"total":107374182400,"available":107345821696,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":48,"seq":206158430232,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27692,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":46,"seq":197568495642,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27692,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":59,"seq":253403070489,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27696,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829904,"statfs":{"total":107374182400,"available":107345821696,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:44:47.650 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-09-10T19:42:55.225356+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=19, tm_min=42, tm_sec=55, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=19, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T19:44:47.651 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-09-10T19:42:55.225356+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=19, tm_min=42, tm_sec=55, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=19, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T19:44:47.651 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-09-10T19:42:55.225356+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=19, tm_min=42, tm_sec=55, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=19, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T19:44:47.651 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-09-10T19:42:55.225356+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=19, tm_min=42, tm_sec=55, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=19, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T19:44:47.651 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-09-10T19:42:55.225356+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=19, tm_min=42, tm_sec=55, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=19, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T19:44:47.651 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-09-10T19:42:55.225356+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=19, tm_min=42, tm_sec=55, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=19, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T19:44:47.651 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-10T19:42:55.225356+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=19, tm_min=42, tm_sec=55, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=19, tm_min=44, tm_sec=42, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T19:44:47.651 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-10T19:45:07.653 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T19:45:07.946 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-10T19:45:07.947 INFO:teuthology.orchestra.run.smithi082.stderr:dumped all 2024-09-10T19:45:07.959 INFO:teuthology.orchestra.run.smithi082.stdout:{"pg_ready":true,"pg_map":{"version":118,"stamp":"2024-09-10T19:45:07.030643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":276904,"kb_used_data":6648,"kb_used_omap":15,"kb_used_meta":270064,"kb_avail":1048299096,"statfs":{"total":1073741824000,"available":1073458274304,"internally_reserved":0,"allocated":6807552,"data_stored":3302178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":276545973},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002814"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":84,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:45.234302+0000","last_change":"2024-09-10T19:44:45.234302+0000","last_active":"2024-09-10T19:44:45.234302+0000","last_peered":"2024-09-10T19:44:45.234302+0000","last_clean":"2024-09-10T19:44:45.234302+0000","last_became_active":"2024-09-10T19:44:22.570915+0000","last_became_peered":"2024-09-10T19:44:22.570915+0000","last_unstale":"2024-09-10T19:44:45.234302+0000","last_undegraded":"2024-09-10T19:44:45.234302+0000","last_fullsized":"2024-09-10T19:44:45.234302+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:44:45.234248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:44:45.234248+0000","last_clean_scrub_stamp":"2024-09-10T19:44:45.234248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:26:37.718433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047593899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":104,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:46.209285+0000","last_change":"2024-09-10T19:44:46.209285+0000","last_active":"2024-09-10T19:44:46.209285+0000","last_peered":"2024-09-10T19:44:46.209285+0000","last_clean":"2024-09-10T19:44:46.209285+0000","last_became_active":"2024-09-10T19:44:22.573940+0000","last_became_peered":"2024-09-10T19:44:22.573940+0000","last_unstale":"2024-09-10T19:44:46.209285+0000","last_undegraded":"2024-09-10T19:44:46.209285+0000","last_fullsized":"2024-09-10T19:44:46.209285+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:44:46.209233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:44:46.209233+0000","last_clean_scrub_stamp":"2024-09-10T19:44:46.209233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:55:32.905521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041823700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":116,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:47.772709+0000","last_change":"2024-09-10T19:44:47.772709+0000","last_active":"2024-09-10T19:44:47.772709+0000","last_peered":"2024-09-10T19:44:47.772709+0000","last_clean":"2024-09-10T19:44:47.772709+0000","last_became_active":"2024-09-10T19:44:36.722538+0000","last_became_peered":"2024-09-10T19:44:36.722538+0000","last_unstale":"2024-09-10T19:44:47.772709+0000","last_undegraded":"2024-09-10T19:44:47.772709+0000","last_fullsized":"2024-09-10T19:44:47.772709+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:44:47.772660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:44:47.772660+0000","last_clean_scrub_stamp":"2024-09-10T19:44:47.772660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:31:36.277742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001104196,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":88,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:43.921844+0000","last_change":"2024-09-10T19:44:43.921844+0000","last_active":"2024-09-10T19:44:43.921844+0000","last_peered":"2024-09-10T19:44:43.921844+0000","last_clean":"2024-09-10T19:44:43.921844+0000","last_became_active":"2024-09-10T19:44:37.084491+0000","last_became_peered":"2024-09-10T19:44:37.084491+0000","last_unstale":"2024-09-10T19:44:43.921844+0000","last_undegraded":"2024-09-10T19:44:43.921844+0000","last_fullsized":"2024-09-10T19:44:43.921844+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:44:43.921783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:44:43.921783+0000","last_clean_scrub_stamp":"2024-09-10T19:44:43.921783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:27:47.671188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042875399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"60'2","reported_seq":107,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:46.222731+0000","last_change":"2024-09-10T19:44:46.222731+0000","last_active":"2024-09-10T19:44:46.222731+0000","last_peered":"2024-09-10T19:44:46.222731+0000","last_clean":"2024-09-10T19:44:46.222731+0000","last_became_active":"2024-09-10T19:44:36.611774+0000","last_became_peered":"2024-09-10T19:44:36.611774+0000","last_unstale":"2024-09-10T19:44:46.222731+0000","last_undegraded":"2024-09-10T19:44:46.222731+0000","last_fullsized":"2024-09-10T19:44:46.222731+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-10T19:44:46.222660+0000","last_deep_scrub":"60'2","last_deep_scrub_stamp":"2024-09-10T19:44:46.222660+0000","last_clean_scrub_stamp":"2024-09-10T19:44:46.222660+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:13:29.487792+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00092825199999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"60'3","reported_seq":113,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:45.268405+0000","last_change":"2024-09-10T19:44:45.268405+0000","last_active":"2024-09-10T19:44:45.268405+0000","last_peered":"2024-09-10T19:44:45.268405+0000","last_clean":"2024-09-10T19:44:45.268405+0000","last_became_active":"2024-09-10T19:44:36.611940+0000","last_became_peered":"2024-09-10T19:44:36.611940+0000","last_unstale":"2024-09-10T19:44:45.268405+0000","last_undegraded":"2024-09-10T19:44:45.268405+0000","last_fullsized":"2024-09-10T19:44:45.268405+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3","last_scrub_stamp":"2024-09-10T19:44:45.268360+0000","last_deep_scrub":"60'3","last_deep_scrub_stamp":"2024-09-10T19:44:45.268360+0000","last_clean_scrub_stamp":"2024-09-10T19:44:45.268360+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:51:24.027438+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00084709599999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":82,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:46.658567+0000","last_change":"2024-09-10T19:44:46.658567+0000","last_active":"2024-09-10T19:44:46.658567+0000","last_peered":"2024-09-10T19:44:46.658567+0000","last_clean":"2024-09-10T19:44:46.658567+0000","last_became_active":"2024-09-10T19:44:22.571306+0000","last_became_peered":"2024-09-10T19:44:22.571306+0000","last_unstale":"2024-09-10T19:44:46.658567+0000","last_undegraded":"2024-09-10T19:44:46.658567+0000","last_fullsized":"2024-09-10T19:44:46.658567+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:44:46.658513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:44:46.658513+0000","last_clean_scrub_stamp":"2024-09-10T19:44:46.658513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:35:07.611095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048965100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":96,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:44:46.028520+0000","last_change":"2024-09-10T19:44:46.028520+0000","last_active":"2024-09-10T19:44:46.028520+0000","last_peered":"2024-09-10T19:44:46.028520+0000","last_clean":"2024-09-10T19:44:46.028520+0000","last_became_active":"2024-09-10T19:44:22.574503+0000","last_became_peered":"2024-09-10T19:44:22.574503+0000","last_unstale":"2024-09-10T19:44:46.028520+0000","last_undegraded":"2024-09-10T19:44:46.028520+0000","last_fullsized":"2024-09-10T19:44:46.028520+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:44:46.028461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:44:46.028461+0000","last_clean_scrub_stamp":"2024-09-10T19:44:46.028461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:22:21.925078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000443783,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":5,"up_from":46,"seq":197568495647,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829912,"statfs":{"total":107374182400,"available":107345829888,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":39,"seq":167503724573,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829912,"statfs":{"total":107374182400,"available":107345829888,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":46,"seq":197568495644,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829912,"statfs":{"total":107374182400,"available":107345829888,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":59,"seq":253403070492,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27692,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":684032,"data_stored":330233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495644,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829912,"statfs":{"total":107374182400,"available":107345829888,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":46,"seq":197568495645,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27692,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":684032,"data_stored":330233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":46,"seq":197568495644,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829912,"statfs":{"total":107374182400,"available":107345829888,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":48,"seq":206158430236,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27692,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":46,"seq":197568495646,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27692,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":59,"seq":253403070493,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27696,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829904,"statfs":{"total":107374182400,"available":107345821696,"internally_reserved":0,"allocated":679936,"data_stored":330214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:45:07.960 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-10T19:45:08.265 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-10T19:45:08.265 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-10T19:45:08.265 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-10T19:45:08.265 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T19:45:08.296 INFO:tasks.ceph.osd.0:Stopped 2024-09-10T19:45:08.296 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-10T19:45:08.296 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T19:45:08.363 INFO:tasks.ceph.osd.1:Stopped 2024-09-10T19:45:08.363 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-10T19:45:08.363 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T19:45:08.431 INFO:tasks.ceph.osd.2:Stopped 2024-09-10T19:45:08.431 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-10T19:45:08.431 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T19:45:08.499 INFO:tasks.ceph.osd.3:Stopped 2024-09-10T19:45:08.499 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-10T19:45:08.499 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T19:45:08.562 INFO:tasks.ceph.osd.4:Stopped 2024-09-10T19:45:08.562 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-10T19:45:08.562 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T19:45:08.632 INFO:tasks.ceph.osd.5:Stopped 2024-09-10T19:45:08.633 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-10T19:45:08.633 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T19:45:08.695 INFO:tasks.ceph.osd.6:Stopped 2024-09-10T19:45:08.695 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-10T19:45:08.695 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T19:45:08.761 INFO:tasks.ceph.osd.7:Stopped 2024-09-10T19:45:08.761 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2024-09-10T19:45:08.761 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T19:45:08.834 INFO:tasks.ceph.osd.8:Stopped 2024-09-10T19:45:08.834 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2024-09-10T19:45:08.834 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T19:45:08.905 INFO:tasks.ceph.osd.9:Stopped 2024-09-10T19:45:08.905 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-10T19:45:08.906 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-10T19:45:08.906 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T19:45:08.971 INFO:tasks.ceph.mgr.x:Stopped 2024-09-10T19:45:08.971 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-10T19:45:08.972 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-10T19:45:08.972 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T19:45:09.007 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-09-10T19:45:09.007 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-09-10T19:45:09.007 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-09-10T19:45:09.007 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.8 is failed for ~0s 2024-09-10T19:45:09.007 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~0s 2024-09-10T19:45:09.023 INFO:tasks.ceph.mon.a:Stopped 2024-09-10T19:45:09.023 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-10T19:45:09.023 DEBUG:teuthology.orchestra.run.smithi082:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-09-10T19:45:09.090 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi082.front.sepia.ceph.com 2024-09-10T19:45:09.091 DEBUG:teuthology.orchestra.run.smithi082:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-10T19:45:09.314 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi082.front.sepia.ceph.com 2024-09-10T19:45:09.315 DEBUG:teuthology.orchestra.run.smithi082:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-10T19:45:09.379 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi082.front.sepia.ceph.com 2024-09-10T19:45:09.379 DEBUG:teuthology.orchestra.run.smithi082:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-10T19:45:09.447 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi082.front.sepia.ceph.com 2024-09-10T19:45:09.448 DEBUG:teuthology.orchestra.run.smithi082:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-10T19:45:09.520 INFO:tasks.ceph:Archiving mon data... 2024-09-10T19:45:09.521 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899503/data/mon.a.tgz 2024-09-10T19:45:09.521 DEBUG:teuthology.orchestra.run.smithi082:> mktemp 2024-09-10T19:45:09.536 INFO:teuthology.orchestra.run.smithi082.stdout:/tmp/tmp.qjWX8rs4O7 2024-09-10T19:45:09.536 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.qjWX8rs4O7 2024-09-10T19:45:09.652 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0666 /tmp/tmp.qjWX8rs4O7 2024-09-10T19:45:09.738 DEBUG:teuthology.orchestra.remote:smithi082:/tmp/tmp.qjWX8rs4O7 is 213KB 2024-09-10T19:45:09.833 DEBUG:teuthology.orchestra.run.smithi082:> rm -fr /tmp/tmp.qjWX8rs4O7 2024-09-10T19:45:09.848 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-10T19:45:09.848 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-10T19:45:09.967 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-10T19:45:09.967 INFO:tasks.ceph:Archiving crash dumps... 2024-09-10T19:45:09.968 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899503/remote/smithi082/crash 2024-09-10T19:45:09.969 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-10T19:45:10.032 INFO:tasks.ceph:Compressing logs... 2024-09-10T19:45:10.032 DEBUG:teuthology.orchestra.run.smithi082:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-10T19:45:10.096 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-10T19:45:10.137 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48226.log 2024-09-10T19:45:10.137 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46961.log 2024-09-10T19:45:10.137 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.48226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46638.log 2024-09-10T19:45:10.137 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48226.log.gz 2024-09-10T19:45:10.137 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46961.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.48184.log -- replaced with /var/log/ceph/ceph-client.admin.46961.log.gz 2024-09-10T19:45:10.138 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-09-10T19:45:10.138 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47156.log 2024-09-10T19:45:10.138 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46638.log.gz 2024-09-10T19:45:10.138 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.48184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36889.log 2024-09-10T19:45:10.138 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48184.log.gz 2024-09-10T19:45:10.138 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.47156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45038.log 2024-09-10T19:45:10.138 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47156.log.gz 2024-09-10T19:45:10.138 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.36889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47680.log 2024-09-10T19:45:10.138 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36889.log.gz 2024-09-10T19:45:10.138 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.45038.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27149.log 2024-09-10T19:45:10.138 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45038.log.gz 2024-09-10T19:45:10.138 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.47680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36426.log 2024-09-10T19:45:10.138 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47680.log.gz 2024-09-10T19:45:10.139 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph.tmp-client.admin.27149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45402.log 0.0% 2024-09-10T19:45:10.139 INFO:teuthology.orchestra.run.smithi082.stderr: -- replaced with /var/log/ceph/ceph.tmp-client.admin.27149.log.gz 2024-09-10T19:45:10.139 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.36426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47890.log 2024-09-10T19:45:10.139 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36426.log.gz 2024-09-10T19:45:10.139 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.45402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47470.log 2024-09-10T19:45:10.139 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45402.log.gz 2024-09-10T19:45:10.139 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.47890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45063.log 2024-09-10T19:45:10.139 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47890.log.gz 2024-09-10T19:45:10.139 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.47470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48352.log 2024-09-10T19:45:10.139 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47470.log.gz 2024-09-10T19:45:10.139 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.45063.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-10T19:45:10.139 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45063.log.gz 2024-09-10T19:45:10.139 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.48352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47082.log 2024-09-10T19:45:10.139 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48352.log.gz 2024-09-10T19:45:10.139 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46761.log 2024-09-10T19:45:10.140 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.47082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37100.log 2024-09-10T19:45:10.140 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47082.log.gz 2024-09-10T19:45:10.140 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46761.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46761.log.gz 2024-09-10T19:45:10.140 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44680.log 2024-09-10T19:45:10.140 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.37100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48310.log 2024-09-10T19:45:10.140 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37100.log.gz 2024-09-10T19:45:10.140 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.44680.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.45151.log -- replaced with /var/log/ceph/ceph-client.admin.44680.log.gz 2024-09-10T19:45:10.140 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-09-10T19:45:10.140 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.48310.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-10T19:45:10.140 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48310.log.gz 2024-09-10T19:45:10.140 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.45151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44986.log 2024-09-10T19:45:10.140 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45151.log.gz 2024-09-10T19:45:10.141 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44461.log 2024-09-10T19:45:10.141 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.44986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46219.log 0.0% 2024-09-10T19:45:10.141 INFO:teuthology.orchestra.run.smithi082.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44986.log.gz 2024-09-10T19:45:10.141 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.44461.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44461.log.gz 2024-09-10T19:45:10.141 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48562.log 2024-09-10T19:45:10.141 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2024-09-10T19:45:10.141 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46219.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46219.log.gz 2024-09-10T19:45:10.141 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.48562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44672.log 2024-09-10T19:45:10.141 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48562.log.gz 2024-09-10T19:45:10.141 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46317.log 2024-09-10T19:45:10.141 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.44672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45490.log 2024-09-10T19:45:10.141 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44672.log.gz 2024-09-10T19:45:10.141 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46810.log 2024-09-10T19:45:10.141 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46317.log.gz 2024-09-10T19:45:10.141 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.45490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36849.log 2024-09-10T19:45:10.141 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45490.log.gz 2024-09-10T19:45:10.142 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46712.log 2024-09-10T19:45:10.142 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46810.log.gz 2024-09-10T19:45:10.142 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.36849.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36849.log.gz 2024-09-10T19:45:10.142 INFO:teuthology.orchestra.run.smithi082.stderr: --verbose -- /var/log/ceph/ceph-client.admin.45581.log 2024-09-10T19:45:10.142 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46712.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46712.log.gz 2024-09-10T19:45:10.142 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45050.log 2024-09-10T19:45:10.142 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.45581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46785.log 2024-09-10T19:45:10.142 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45581.log.gz 2024-09-10T19:45:10.142 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.45050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44335.log 2024-09-10T19:45:10.142 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45050.log.gz 2024-09-10T19:45:10.142 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36980.log 2024-09-10T19:45:10.142 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46785.log.gz 2024-09-10T19:45:10.142 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.44335.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.44335.log.gz /var/log/ceph/ceph-client.admin.45060.log 2024-09-10T19:45:10.142 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-09-10T19:45:10.143 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.36980.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36980.log.gz 2024-09-10T19:45:10.143 INFO:teuthology.orchestra.run.smithi082.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47106.log 2024-09-10T19:45:10.143 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.45060.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45060.log.gz 2024-09-10T19:45:10.143 INFO:teuthology.orchestra.run.smithi082.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47974.log 2024-09-10T19:45:10.143 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.47106.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47106.log.gz 2024-09-10T19:45:10.143 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48142.log 2024-09-10T19:45:10.143 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.47974.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.47974.log.gz -5 2024-09-10T19:45:10.143 INFO:teuthology.orchestra.run.smithi082.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44664.log 2024-09-10T19:45:10.143 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.48142.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48142.log.gz 2024-09-10T19:45:10.143 INFO:teuthology.orchestra.run.smithi082.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-10T19:45:10.144 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.44664.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44664.log.gz 2024-09-10T19:45:10.144 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46662.log 2024-09-10T19:45:10.144 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44259.log 2024-09-10T19:45:10.144 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46662.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46662.log.gz 2024-09-10T19:45:10.144 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47058.log 2024-09-10T19:45:10.144 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.44259.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44259.log.gz 2024-09-10T19:45:10.144 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48268.log 2024-09-10T19:45:10.144 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.47058.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47058.log.gz 2024-09-10T19:45:10.144 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46095.log 2024-09-10T19:45:10.144 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.48268.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48268.log.gz 2024-09-10T19:45:10.144 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-10T19:45:10.144 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46095.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46095.log.gz 2024-09-10T19:45:10.144 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46171.log 2024-09-10T19:45:10.144 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46835.log 2024-09-10T19:45:10.145 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46171.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45491.log 2024-09-10T19:45:10.145 INFO:teuthology.orchestra.run.smithi082.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46171.log.gz 2024-09-10T19:45:10.145 INFO:teuthology.orchestra.run.smithi082.stderr: 89.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44377.log 2024-09-10T19:45:10.145 INFO:teuthology.orchestra.run.smithi082.stderr: -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-10T19:45:10.145 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46835.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46835.log.gz 2024-09-10T19:45:10.145 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.45491.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45491.log.gz 2024-09-10T19:45:10.145 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46342.log 2024-09-10T19:45:10.145 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47230.log 2024-09-10T19:45:10.145 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.44377.log: /var/log/ceph/ceph-client.admin.46342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45805.log 2024-09-10T19:45:10.145 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46342.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44377.log.gz 2024-09-10T19:45:10.145 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-09-10T19:45:10.145 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.47230.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47230.log.gz 2024-09-10T19:45:10.145 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45110.log 2024-09-10T19:45:10.145 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45847.log 2024-09-10T19:45:10.145 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.45805.log: /var/log/ceph/ceph-client.admin.45110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46563.log 2024-09-10T19:45:10.146 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45805.log.gz 2024-09-10T19:45:10.146 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45110.log.gz 2024-09-10T19:45:10.146 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.45847.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45847.log.gz 2024-09-10T19:45:10.146 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36769.log 2024-09-10T19:45:10.146 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36524.log 2024-09-10T19:45:10.146 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46563.log: /var/log/ceph/ceph-client.admin.36769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46986.log 2024-09-10T19:45:10.146 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46563.log.gz 2024-09-10T19:45:10.146 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36769.log.gz 2024-09-10T19:45:10.146 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.36524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45484.log 2024-09-10T19:45:10.146 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36524.log.gz 2024-09-10T19:45:10.146 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48394.log 2024-09-10T19:45:10.146 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46986.log: /var/log/ceph/ceph-client.admin.45484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44284.log 2024-09-10T19:45:10.146 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46986.log.gz 2024-09-10T19:45:10.146 INFO:teuthology.orchestra.run.smithi082.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45484.log.gz 2024-09-10T19:45:10.146 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.48394.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48394.log.gz 2024-09-10T19:45:10.147 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46195.log 2024-09-10T19:45:10.147 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36940.log 2024-09-10T19:45:10.147 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.44284.log: /var/log/ceph/ceph-client.admin.46195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44667.log 2024-09-10T19:45:10.147 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44284.log.gz 2024-09-10T19:45:10.147 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46195.log.gz 2024-09-10T19:45:10.147 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.36940.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36940.log.gz 2024-09-10T19:45:10.147 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44674.log 2024-09-10T19:45:10.147 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46513.log 2024-09-10T19:45:10.147 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.44667.log: /var/log/ceph/ceph-client.admin.44674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37376.log 2024-09-10T19:45:10.147 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44667.log.gz 2024-09-10T19:45:10.147 INFO:teuthology.orchestra.run.smithi082.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44674.log.gz 2024-09-10T19:45:10.147 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46513.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46513.log.gz 2024-09-10T19:45:10.147 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37140.log 2024-09-10T19:45:10.147 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47010.log 2024-09-10T19:45:10.147 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.37376.log: /var/log/ceph/ceph-client.admin.37140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47512.log 2024-09-10T19:45:10.147 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37376.log.gz 2024-09-10T19:45:10.148 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37140.log.gz 2024-09-10T19:45:10.148 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.47010.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47010.log.gz 2024-09-10T19:45:10.148 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47848.log 2024-09-10T19:45:10.148 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44419.log 2024-09-10T19:45:10.148 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.47512.log: /var/log/ceph/ceph-client.admin.47848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48058.log 2024-09-10T19:45:10.148 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47512.log.gz 2024-09-10T19:45:10.148 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47848.log.gz 2024-09-10T19:45:10.148 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.44419.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44419.log.gz 2024-09-10T19:45:10.148 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45051.log 2024-09-10T19:45:10.148 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37573.log 2024-09-10T19:45:10.148 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.48058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47932.log 2024-09-10T19:45:10.148 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.45051.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48058.log.gz 2024-09-10T19:45:10.149 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45051.log.gz 2024-09-10T19:45:10.149 INFO:teuthology.orchestra.run.smithi082.stderr:gzip/var/log/ceph/ceph-client.admin.37573.log: -5 --verbose -- /var/log/ceph/ceph-mon.a.log 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37573.log.gz 2024-09-10T19:45:10.149 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-09-10T19:45:10.149 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.47932.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47932.log.gz 2024-09-10T19:45:10.149 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47131.log 2024-09-10T19:45:10.149 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45047.log 2024-09-10T19:45:10.149 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.47131.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46737.log 2024-09-10T19:45:10.149 INFO:teuthology.orchestra.run.smithi082.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47131.log.gz 2024-09-10T19:45:10.149 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.45047.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45047.log.gz 2024-09-10T19:45:10.149 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47344.log 2024-09-10T19:45:10.149 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46737.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46737.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48016.log 2024-09-10T19:45:10.149 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-09-10T19:45:10.149 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46440.log 2024-09-10T19:45:10.149 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.47344.log: /var/log/ceph/ceph-client.admin.48016.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48016.log.gz 2024-09-10T19:45:10.150 INFO:teuthology.orchestra.run.smithi082.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47344.log.gz 2024-09-10T19:45:10.150 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45343.log 2024-09-10T19:45:10.150 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46440.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46440.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47596.log 2024-09-10T19:45:10.150 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-09-10T19:45:10.150 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46488.log 2024-09-10T19:45:10.150 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.45343.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45343.log.gz 2024-09-10T19:45:10.151 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.47596.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47596.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-10T19:45:10.151 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-09-10T19:45:10.151 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45763.log 2024-09-10T19:45:10.151 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.46488.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46488.log.gz 2024-09-10T19:45:10.151 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47034.log 2024-09-10T19:45:10.151 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.45763.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45763.log.gz 2024-09-10T19:45:10.151 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47181.log 2024-09-10T19:45:10.152 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.47034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47034.log.gz 2024-09-10T19:45:10.152 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-10T19:45:10.152 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.47181.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47181.log.gz 2024-09-10T19:45:10.152 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-10T19:45:10.153 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46464.log 2024-09-10T19:45:10.164 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-osd.5.log: 91.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-10T19:45:10.164 INFO:teuthology.orchestra.run.smithi082.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-10T19:45:10.167 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-10T19:45:10.211 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46464.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46464.log.gz 2024-09-10T19:45:10.211 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47806.log 2024-09-10T19:45:10.211 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph.log: 85.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-10T19:45:10.211 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44203.log 2024-09-10T19:45:10.212 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.47806.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47806.log.gz 2024-09-10T19:45:10.212 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37060.log 2024-09-10T19:45:10.212 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.44203.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44203.log.gz 2024-09-10T19:45:10.212 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48478.log 2024-09-10T19:45:10.212 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.37060.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37060.log.gz 2024-09-10T19:45:10.212 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46687.log 2024-09-10T19:45:10.212 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.48478.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48478.log.gz 2024-09-10T19:45:10.212 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46267.log 2024-09-10T19:45:10.212 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46687.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46687.log.gz 2024-09-10T19:45:10.212 INFO:teuthology.orchestra.run.smithi082.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-10T19:45:10.212 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46909.log 2024-09-10T19:45:10.212 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-10T19:45:10.212 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46267.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46267.log.gz 2024-09-10T19:45:10.213 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46909.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.46909.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.37020.log 2024-09-10T19:45:10.213 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-09-10T19:45:10.213 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-mgr.x.log: 92.1% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2024-09-10T19:45:10.213 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36809.log 2024-09-10T19:45:10.213 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.37020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44675.log 2024-09-10T19:45:10.213 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37020.log.gz 2024-09-10T19:45:10.213 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.36809.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36809.log.gz 2024-09-10T19:45:10.213 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45585.log 2024-09-10T19:45:10.213 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.44675.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44675.log.gz 2024-09-10T19:45:10.213 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48520.log 2024-09-10T19:45:10.213 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.45585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48100.log 2024-09-10T19:45:10.213 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45585.log.gz 2024-09-10T19:45:10.213 INFO:teuthology.orchestra.run.smithi082.stderr: 92.3%/var/log/ceph/ceph-client.admin.48520.log: -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-10T19:45:10.213 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48520.log.gz 2024-09-10T19:45:10.214 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46538.log 2024-09-10T19:45:10.214 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.48100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47428.log 2024-09-10T19:45:10.214 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48100.log.gz 2024-09-10T19:45:10.214 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46538.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.8.log 2024-09-10T19:45:10.214 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46538.log.gz 2024-09-10T19:45:10.214 INFO:teuthology.orchestra.run.smithi082.stderr:gzip/var/log/ceph/ceph-client.admin.47428.log: -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.46119.log 2024-09-10T19:45:10.214 INFO:teuthology.orchestra.run.smithi082.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47428.log.gz 2024-09-10T19:45:10.214 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46367.log 2024-09-10T19:45:10.214 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46119.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46119.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45403.log 2024-09-10T19:45:10.214 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-09-10T19:45:10.214 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44683.log 2024-09-10T19:45:10.214 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46367.log: /var/log/ceph/ceph-client.admin.45403.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45403.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46367.log.gz 2024-09-10T19:45:10.214 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-09-10T19:45:10.214 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44666.log 2024-09-10T19:45:10.215 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47638.log 2024-09-10T19:45:10.215 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.44683.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44683.log.gz 2024-09-10T19:45:10.215 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.44666.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44666.log.gz 2024-09-10T19:45:10.215 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46860.log 2024-09-10T19:45:10.215 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44678.log 2024-09-10T19:45:10.215 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.47638.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47638.log.gz 2024-09-10T19:45:10.215 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46860.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46860.log.gz 2024-09-10T19:45:10.215 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47722.log 2024-09-10T19:45:10.215 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48602.log 2024-09-10T19:45:10.215 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.44678.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44678.log.gz 2024-09-10T19:45:10.215 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.47722.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47722.log.gz 2024-09-10T19:45:10.215 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45056.log 2024-09-10T19:45:10.215 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.48602.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48602.log.gz 2024-09-10T19:45:10.216 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44679.log 2024-09-10T19:45:10.216 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45401.log 2024-09-10T19:45:10.216 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.45056.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45056.log.gz 2024-09-10T19:45:10.216 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.44679.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44679.log.gz 2024-09-10T19:45:10.216 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46588.log 2024-09-10T19:45:10.216 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.45401.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45401.log.gz 2024-09-10T19:45:10.216 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46243.log 2024-09-10T19:45:10.216 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.46588.log: -- /var/log/ceph/ceph-client.admin.46391.log 2024-09-10T19:45:10.216 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46588.log.gz 2024-09-10T19:45:10.216 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47206.log 2024-09-10T19:45:10.216 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46243.log.gz 2024-09-10T19:45:10.216 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46391.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46391.log.gz 2024-09-10T19:45:10.216 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48436.log 2024-09-10T19:45:10.217 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.47206.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47206.log.gz 92.2%gzip -- replaced with /var/log/ceph/ceph-osd.4.log.gz -5 2024-09-10T19:45:10.217 INFO:teuthology.orchestra.run.smithi082.stderr: --verbose -- /var/log/ceph/ceph-client.admin.45493.log 2024-09-10T19:45:10.217 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-09-10T19:45:10.217 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47554.log 2024-09-10T19:45:10.217 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.48436.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48436.log.gz/var/log/ceph/ceph-client.admin.45493.log: 2024-09-10T19:45:10.217 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46067.log 2024-09-10T19:45:10.217 INFO:teuthology.orchestra.run.smithi082.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45493.log.gz 2024-09-10T19:45:10.217 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5/var/log/ceph/ceph-client.admin.47554.log: --verbose -- /var/log/ceph/ceph-osd.3.log 0.0% 2024-09-10T19:45:10.217 INFO:teuthology.orchestra.run.smithi082.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47554.log.gz 2024-09-10T19:45:10.217 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46067.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46067.log.gz 2024-09-10T19:45:10.217 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46613.log 2024-09-10T19:45:10.217 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose/var/log/ceph/ceph-osd.3.log: -- /var/log/ceph/ceph-client.admin.47386.log 2024-09-10T19:45:10.217 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46613.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46613.log.gz 2024-09-10T19:45:10.218 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46416.log 2024-09-10T19:45:10.218 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.47386.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47764.log 2024-09-10T19:45:10.218 INFO:teuthology.orchestra.run.smithi082.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47386.log.gz 2024-09-10T19:45:10.218 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46885.log 2024-09-10T19:45:10.218 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46416.log: /var/log/ceph/ceph-client.admin.47764.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46416.log.gz 2024-09-10T19:45:10.218 INFO:teuthology.orchestra.run.smithi082.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47764.log.gz 2024-09-10T19:45:10.218 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46935.log 2024-09-10T19:45:10.218 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46292.log 2024-09-10T19:45:10.218 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46885.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46885.log.gz 2024-09-10T19:45:10.218 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46935.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46935.log.gz 2024-09-10T19:45:10.218 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.46292.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46292.log.gz 2024-09-10T19:45:10.218 INFO:teuthology.orchestra.run.smithi082.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-10T19:45:10.219 INFO:teuthology.orchestra.run.smithi082.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-10T19:45:10.219 INFO:teuthology.orchestra.run.smithi082.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-10T19:45:10.225 INFO:teuthology.orchestra.run.smithi082.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2024-09-10T19:45:10.248 INFO:teuthology.orchestra.run.smithi082.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-10T19:45:10.249 INFO:teuthology.orchestra.run.smithi082.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-10T19:45:10.251 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-09-10T19:45:10.251 INFO:teuthology.orchestra.run.smithi082.stderr:real 0m0.164s 2024-09-10T19:45:10.252 INFO:teuthology.orchestra.run.smithi082.stderr:user 0m0.806s 2024-09-10T19:45:10.252 INFO:teuthology.orchestra.run.smithi082.stderr:sys 0m0.114s 2024-09-10T19:45:10.252 INFO:tasks.ceph:Archiving logs... 2024-09-10T19:45:10.252 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899503/remote/smithi082/log 2024-09-10T19:45:10.253 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-10T19:45:10.515 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-10T19:45:10.532 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-10T19:45:10.532 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-10T19:45:10.582 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-10T19:45:10.590 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-10T19:45:10.590 DEBUG:teuthology.orchestra.run.smithi082:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T19:45:10.612 INFO:teuthology.orchestra.run.smithi082.stderr:bash: line 1: ntpq: command not found 2024-09-10T19:45:10.617 INFO:teuthology.orchestra.run.smithi082.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T19:45:10.618 INFO:teuthology.orchestra.run.smithi082.stdout:=============================================================================== 2024-09-10T19:45:10.618 INFO:teuthology.orchestra.run.smithi082.stdout:^* hv01.front.sepia.ceph.com 3 6 377 45 -586us[ -595us] +/- 58ms 2024-09-10T19:45:10.618 INFO:teuthology.orchestra.run.smithi082.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 43 +1913us[+1913us] +/- 61ms 2024-09-10T19:45:10.618 INFO:teuthology.orchestra.run.smithi082.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 45 -1590us[-1590us] +/- 74ms 2024-09-10T19:45:10.618 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T19:45:10.618 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-10T19:45:10.626 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-10T19:45:10.626 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-10T19:45:10.669 DEBUG:teuthology.orchestra.run.smithi082:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-10T19:45:10.731 DEBUG:teuthology.orchestra.run.smithi082:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T19:45:10.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:45:10.758 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-10T19:45:10.767 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-10T19:45:10.794 INFO:teuthology.task.internal:Duration was 529.125355 seconds 2024-09-10T19:45:10.794 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-10T19:45:10.844 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-10T19:45:10.844 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-10T19:45:10.882 INFO:teuthology.orchestra.run.smithi082.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T19:45:11.172 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-10T19:45:11.172 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi082.front.sepia.ceph.com 2024-09-10T19:45:11.172 DEBUG:teuthology.orchestra.run.smithi082:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-10T19:45:11.232 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-10T19:45:11.232 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-10T19:45:11.298 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-10T19:45:11.298 DEBUG:teuthology.orchestra.run.smithi082:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-10T19:45:11.440 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-10T19:45:11.449 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-10T19:45:11.449 DEBUG:teuthology.orchestra.run.smithi082:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-10T19:45:11.473 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-10T19:45:11.494 DEBUG:teuthology.orchestra.run.smithi082:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-10T19:45:11.536 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = core 2024-09-10T19:45:11.549 DEBUG:teuthology.orchestra.run.smithi082:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-10T19:45:11.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:45:11.609 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-10T19:45:11.621 INFO:teuthology.task.internal:Transferring archived files... 2024-09-10T19:45:11.621 DEBUG:teuthology.misc:Transferring archived files from smithi082:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899503/remote/smithi082 2024-09-10T19:45:11.622 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-10T19:45:11.681 INFO:teuthology.task.internal:Removing archive directory... 2024-09-10T19:45:11.681 DEBUG:teuthology.orchestra.run.smithi082:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-10T19:45:11.731 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-10T19:45:11.740 INFO:teuthology.task.internal:Not uploading archives. 2024-09-10T19:45:11.740 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-10T19:45:11.770 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-10T19:45:11.771 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-10T19:45:11.786 INFO:teuthology.orchestra.run.smithi082.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 10 19:45 /home/ubuntu/cephtest 2024-09-10T19:45:11.787 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-10T19:45:11.820 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-10T19:45:11.922 INFO:teuthology.run:Summary data: description: rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{centos_latest}} duration: 529.1253547668457 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-10T19:45:11.922 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-10T19:45:11.991 INFO:teuthology.run:pass