2024-09-11T14:30:34.873 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-11T14:30:34.891 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T14:30:34.974 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901371 branch: main description: crimson-rados/thrash_simple/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore thrashers/simple thrashosds-health workloads/admin_socket_objecter_requests} email: null first_in_suite: false job_id: '7901371' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client.0: admin socket: /var/run/ceph/ceph-$name.asok global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 osd pool default crimson: true paxos service trim min: 10 osd: crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug ms: 1 debug osd: 20 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore ompap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd max markdown count: 1000 osd objectstore: seastore osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - SLOW_OPS - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub sha1: 0d61375b8ab53be4c233f5b93e463edfdff9316a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 0d61375b8ab53be4c233f5b93e463edfdff9316a thrashosds: min_in: 4 workunit: branch: wip-matanb-crimson-seastore-logs env: CRIMSON_COMPAT: '1' sha1: dbebf1a3db6db7b5579464ee715edcf1b3e9cce2 owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - osd.0 - osd.1 - client.0 - node-exporter.a - - mgr.x - osd.2 - osd.3 - client.1 - prometheus.a - node-exporter.b seed: 5813 sha1: 0d61375b8ab53be4c233f5b93e463edfdff9316a sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-matanb-crimson-seastore-logs suite_path: /home/teuthworker/src/github.com_matan-b_ceph_dbebf1a3db6db7b5579464ee715edcf1b3e9cce2/qa suite_relpath: qa suite_repo: https://github.com/matan-b/ceph.git suite_sha1: dbebf1a3db6db7b5579464ee715edcf1b3e9cce2 targets: smithi181.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM8vMyk6v77e2SDon6qWKITXhlQsrMYSD1l9ELy92Nnm9akngN9aF1FVhFjl8cbtFSeyNFSUJxSjp8vvoOlU/us= smithi203.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM07U4gC4pLpBCpPvX+PPOBwSdNsLUuGHL5qCO+OrSXNZhNOI8SQqR3nPk4x8ZIurmHPghFk9t8LeDtNlzKZj8E= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson - thrashosds: ceph_objectstore_tool: false chance_down: 0 chance_inject_pause_short: 0 chance_reset_purged_snaps_last: 0 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 dump_ops_enable: false min_in: 3 noscrub_toggle_delay: 0 reweight_osd: 0 sighup_delay: 0 thrash_primary_affinity: false timeout: 2400 - radosbench: clients: - client.0 time: 150 - admin_socket: client.0: objecter_requests: test: http://git.ceph.com/?p={repo};a=blob_plain;f=src/test/admin_socket/objecter_requests;hb={branch} teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-11_13:56:22 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-11T14:30:34.974 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_matan-b_ceph_dbebf1a3db6db7b5579464ee715edcf1b3e9cce2/qa; will attempt to use it 2024-09-11T14:30:34.975 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_matan-b_ceph_dbebf1a3db6db7b5579464ee715edcf1b3e9cce2/qa/tasks 2024-09-11T14:30:34.975 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-11T14:30:34.977 INFO:teuthology.task.internal:Checking packages... 2024-09-11T14:30:34.998 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '0d61375b8ab53be4c233f5b93e463edfdff9316a' 2024-09-11T14:30:34.998 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-11T14:30:34.998 INFO:teuthology.packaging:ref: None 2024-09-11T14:30:34.998 INFO:teuthology.packaging:tag: None 2024-09-11T14:30:34.998 INFO:teuthology.packaging:branch: main 2024-09-11T14:30:34.998 INFO:teuthology.packaging:sha1: 0d61375b8ab53be4c233f5b93e463edfdff9316a 2024-09-11T14:30:34.998 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=main 2024-09-11T14:30:35.203 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4872.g0d61375b 2024-09-11T14:30:35.204 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-11T14:30:35.214 INFO:teuthology.task.internal:no buildpackages task found 2024-09-11T14:30:35.214 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-11T14:30:35.219 INFO:teuthology.task.internal:Saving configuration 2024-09-11T14:30:35.233 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-11T14:30:35.240 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-11T14:30:35.266 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi181.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901371', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 14:25:34.226353', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM8vMyk6v77e2SDon6qWKITXhlQsrMYSD1l9ELy92Nnm9akngN9aF1FVhFjl8cbtFSeyNFSUJxSjp8vvoOlU/us='} 2024-09-11T14:30:35.288 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901371', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 14:25:34.224860', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM07U4gC4pLpBCpPvX+PPOBwSdNsLUuGHL5qCO+OrSXNZhNOI8SQqR3nPk4x8ZIurmHPghFk9t8LeDtNlzKZj8E='} 2024-09-11T14:30:35.289 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-11T14:30:35.295 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'] 2024-09-11T14:30:35.295 INFO:teuthology.task.internal:roles: ubuntu@smithi203.front.sepia.ceph.com - ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-09-11T14:30:35.295 INFO:teuthology.run_tasks:Running task console_log... 2024-09-11T14:30:35.353 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7e28277ac0>, signals=[15]) 2024-09-11T14:30:35.353 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-11T14:30:35.359 INFO:teuthology.task.internal:Opening connections... 2024-09-11T14:30:35.359 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi181.front.sepia.ceph.com 2024-09-11T14:30:35.360 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T14:30:35.437 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi203.front.sepia.ceph.com 2024-09-11T14:30:35.438 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T14:30:35.514 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-11T14:30:35.522 DEBUG:teuthology.orchestra.run.smithi181:> uname -m 2024-09-11T14:30:35.539 INFO:teuthology.orchestra.run.smithi181.stdout:x86_64 2024-09-11T14:30:35.540 DEBUG:teuthology.orchestra.run.smithi181:> cat /etc/os-release 2024-09-11T14:30:35.597 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="CentOS Stream" 2024-09-11T14:30:35.598 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="9" 2024-09-11T14:30:35.598 INFO:teuthology.orchestra.run.smithi181.stdout:ID="centos" 2024-09-11T14:30:35.598 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE="rhel fedora" 2024-09-11T14:30:35.598 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="9" 2024-09-11T14:30:35.598 INFO:teuthology.orchestra.run.smithi181.stdout:PLATFORM_ID="platform:el9" 2024-09-11T14:30:35.598 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-11T14:30:35.598 INFO:teuthology.orchestra.run.smithi181.stdout:ANSI_COLOR="0;31" 2024-09-11T14:30:35.598 INFO:teuthology.orchestra.run.smithi181.stdout:LOGO="fedora-logo-icon" 2024-09-11T14:30:35.598 INFO:teuthology.orchestra.run.smithi181.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-11T14:30:35.598 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://centos.org/" 2024-09-11T14:30:35.598 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-11T14:30:35.598 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-11T14:30:35.598 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-11T14:30:35.599 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server 2024-09-11T14:30:35.616 DEBUG:teuthology.orchestra.run.smithi203:> uname -m 2024-09-11T14:30:35.634 INFO:teuthology.orchestra.run.smithi203.stdout:x86_64 2024-09-11T14:30:35.634 DEBUG:teuthology.orchestra.run.smithi203:> cat /etc/os-release 2024-09-11T14:30:35.691 INFO:teuthology.orchestra.run.smithi203.stdout:NAME="CentOS Stream" 2024-09-11T14:30:35.691 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION="9" 2024-09-11T14:30:35.691 INFO:teuthology.orchestra.run.smithi203.stdout:ID="centos" 2024-09-11T14:30:35.691 INFO:teuthology.orchestra.run.smithi203.stdout:ID_LIKE="rhel fedora" 2024-09-11T14:30:35.691 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_ID="9" 2024-09-11T14:30:35.691 INFO:teuthology.orchestra.run.smithi203.stdout:PLATFORM_ID="platform:el9" 2024-09-11T14:30:35.691 INFO:teuthology.orchestra.run.smithi203.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-11T14:30:35.691 INFO:teuthology.orchestra.run.smithi203.stdout:ANSI_COLOR="0;31" 2024-09-11T14:30:35.691 INFO:teuthology.orchestra.run.smithi203.stdout:LOGO="fedora-logo-icon" 2024-09-11T14:30:35.691 INFO:teuthology.orchestra.run.smithi203.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-11T14:30:35.691 INFO:teuthology.orchestra.run.smithi203.stdout:HOME_URL="https://centos.org/" 2024-09-11T14:30:35.691 INFO:teuthology.orchestra.run.smithi203.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-11T14:30:35.691 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-11T14:30:35.692 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-11T14:30:35.692 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server 2024-09-11T14:30:35.716 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-11T14:30:35.725 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-11T14:30:35.730 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-11T14:30:35.730 DEBUG:teuthology.orchestra.run.smithi181:> test '!' -e /home/ubuntu/cephtest 2024-09-11T14:30:35.733 DEBUG:teuthology.orchestra.run.smithi203:> test '!' -e /home/ubuntu/cephtest 2024-09-11T14:30:35.749 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-11T14:30:35.756 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-11T14:30:35.756 DEBUG:teuthology.orchestra.run.smithi181:> test -z $(ls -A /var/lib/ceph) 2024-09-11T14:30:35.791 DEBUG:teuthology.orchestra.run.smithi203:> test -z $(ls -A /var/lib/ceph) 2024-09-11T14:30:36.021 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-11T14:30:36.076 INFO:teuthology.run_tasks:Running task kernel... 2024-09-11T14:30:36.130 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-09-11T14:30:36.130 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-09-11T14:30:36.130 DEBUG:teuthology.orchestra.run.smithi181:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T14:30:36.131 DEBUG:teuthology.orchestra.run.smithi203:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T14:30:36.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T14:30:36.148 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2024-09-11T14:30:36.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T14:30:36.149 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-09-11T14:30:36.206 INFO:teuthology.orchestra.run.smithi203.stdout:5.14.0-503.el9.x86_64 2024-09-11T14:30:36.206 INFO:teuthology.task.kernel:Running kernel on smithi203: 5.14.0-503.el9.x86_64 2024-09-11T14:30:36.206 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum install -y kernel 2024-09-11T14:30:36.207 INFO:teuthology.orchestra.run.smithi181.stdout:5.14.0-503.el9.x86_64 2024-09-11T14:30:36.207 INFO:teuthology.task.kernel:Running kernel on smithi181: 5.14.0-503.el9.x86_64 2024-09-11T14:30:36.208 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum install -y kernel 2024-09-11T14:30:38.661 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.2 MB 00:00 2024-09-11T14:30:39.009 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - BaseOS 8.1 MB/s | 8.2 MB 00:01 2024-09-11T14:30:40.810 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - AppStream 31 MB/s | 20 MB 00:00 2024-09-11T14:30:47.511 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - CRB 2.3 MB/s | 6.5 MB 00:02 2024-09-11T14:30:48.945 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - Extras packages 81 kB/s | 19 kB 00:00 2024-09-11T14:30:49.699 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 36 MB/s | 23 MB 00:00 2024-09-11T14:30:55.121 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-09-11T14:30:57.086 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-11T14:30:57.087 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-11T14:30:57.172 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T14:30:57.188 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:30:57.188 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-09-11T14:30:57.188 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:30:57.188 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T14:30:57.188 INFO:teuthology.orchestra.run.smithi181.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-11T14:30:57.189 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-11T14:30:57.189 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-11T14:30:57.189 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-11T14:30:57.189 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-11T14:30:57.189 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:30:57.189 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T14:30:57.189 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:30:57.189 INFO:teuthology.orchestra.run.smithi181.stdout:Install 4 Packages 2024-09-11T14:30:57.189 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:30:57.190 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 79 M 2024-09-11T14:30:57.190 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 126 M 2024-09-11T14:30:57.190 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T14:30:57.406 INFO:teuthology.orchestra.run.smithi181.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 233 kB/s | 23 kB 00:00 2024-09-11T14:30:57.790 INFO:teuthology.orchestra.run.smithi181.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-09-11T14:30:58.231 INFO:teuthology.orchestra.run.smithi181.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 37 MB/s | 35 MB 00:00 2024-09-11T14:30:58.434 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - AppStream 1.1 MB/s | 20 MB 00:18 2024-09-11T14:30:58.490 INFO:teuthology.orchestra.run.smithi181.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 26 MB/s | 29 MB 00:01 2024-09-11T14:30:58.490 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:30:58.490 INFO:teuthology.orchestra.run.smithi181.stdout:Total 61 MB/s | 79 MB 00:01 2024-09-11T14:30:59.170 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T14:30:59.498 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T14:30:59.498 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T14:31:00.449 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T14:31:00.450 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T14:31:02.890 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T14:31:03.533 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-11T14:31:03.623 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T14:31:04.751 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 6.5 MB 00:02 2024-09-11T14:31:06.402 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 19 kB 00:00 2024-09-11T14:31:06.410 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T14:31:06.561 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T14:31:07.297 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-11T14:31:11.375 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T14:31:11.486 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:31:12.695 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2024-09-11T14:31:14.676 INFO:teuthology.orchestra.run.smithi203.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-11T14:31:14.677 INFO:teuthology.orchestra.run.smithi203.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-11T14:31:14.765 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-11T14:31:14.777 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:31:14.777 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-09-11T14:31:14.778 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:31:14.778 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-11T14:31:14.778 INFO:teuthology.orchestra.run.smithi203.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-11T14:31:14.778 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-09-11T14:31:14.778 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-11T14:31:14.778 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-11T14:31:14.778 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-11T14:31:14.778 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:31:14.778 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-11T14:31:14.778 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:31:14.778 INFO:teuthology.orchestra.run.smithi203.stdout:Install 4 Packages 2024-09-11T14:31:14.778 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:31:14.779 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 79 M 2024-09-11T14:31:14.779 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 126 M 2024-09-11T14:31:14.779 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-11T14:31:14.896 INFO:teuthology.orchestra.run.smithi203.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 235 kB/s | 23 kB 00:00 2024-09-11T14:31:16.327 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:31:17.105 INFO:teuthology.orchestra.run.smithi203.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 6.8 MB/s | 16 MB 00:02 2024-09-11T14:31:19.088 INFO:teuthology.orchestra.run.smithi203.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 6.8 MB/s | 29 MB 00:04 2024-09-11T14:31:20.346 INFO:teuthology.orchestra.run.smithi203.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 6.2 MB/s | 35 MB 00:05 2024-09-11T14:31:20.347 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:31:20.347 INFO:teuthology.orchestra.run.smithi203.stdout:Total 14 MB/s | 79 MB 00:05 2024-09-11T14:31:21.034 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-11T14:31:21.390 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-11T14:31:21.390 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-11T14:31:22.364 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-11T14:31:22.364 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-11T14:31:24.804 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-11T14:31:25.536 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-11T14:31:25.622 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T14:31:28.419 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T14:31:28.557 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T14:31:33.361 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T14:31:33.400 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:31:38.267 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:31:54.324 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:31:54.365 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:31:54.737 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:31:54.737 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-11T14:31:54.737 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T14:31:54.737 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T14:31:55.584 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:31:55.584 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:31:55.584 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T14:31:55.584 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-11T14:31:55.584 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-11T14:31:55.584 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-11T14:31:55.584 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-11T14:31:55.584 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:31:55.584 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T14:31:55.937 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T14:31:56.024 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T14:31:56.024 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T14:31:56.025 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-11T14:31:56.025 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T14:31:56.025 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T14:31:56.025 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-11T14:31:56.025 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi181.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T14:31:56.025 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T14:31:56.083 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T14:31:56.083 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T14:31:56.084 DEBUG:teuthology.orchestra.run.smithi181:> sudo rpm -qi grub2-tools 2024-09-11T14:31:56.114 INFO:teuthology.orchestra.run.smithi181.stdout:Name : grub2-tools 2024-09-11T14:31:56.115 INFO:teuthology.orchestra.run.smithi181.stdout:Epoch : 1 2024-09-11T14:31:56.115 INFO:teuthology.orchestra.run.smithi181.stdout:Version : 2.06 2024-09-11T14:31:56.115 INFO:teuthology.orchestra.run.smithi181.stdout:Release : 82.el9 2024-09-11T14:31:56.115 INFO:teuthology.orchestra.run.smithi181.stdout:Architecture: x86_64 2024-09-11T14:31:56.115 INFO:teuthology.orchestra.run.smithi181.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-11T14:31:56.115 INFO:teuthology.orchestra.run.smithi181.stdout:Group : Unspecified 2024-09-11T14:31:56.115 INFO:teuthology.orchestra.run.smithi181.stdout:Size : 8274670 2024-09-11T14:31:56.115 INFO:teuthology.orchestra.run.smithi181.stdout:License : GPLv3+ 2024-09-11T14:31:56.116 INFO:teuthology.orchestra.run.smithi181.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-11T14:31:56.116 INFO:teuthology.orchestra.run.smithi181.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-11T14:31:56.116 INFO:teuthology.orchestra.run.smithi181.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-11T14:31:56.116 INFO:teuthology.orchestra.run.smithi181.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-11T14:31:56.116 INFO:teuthology.orchestra.run.smithi181.stdout:Packager : builder@centos.org 2024-09-11T14:31:56.116 INFO:teuthology.orchestra.run.smithi181.stdout:Vendor : CentOS 2024-09-11T14:31:56.116 INFO:teuthology.orchestra.run.smithi181.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-11T14:31:56.116 INFO:teuthology.orchestra.run.smithi181.stdout:Summary : Support tools for GRUB. 2024-09-11T14:31:56.116 INFO:teuthology.orchestra.run.smithi181.stdout:Description : 2024-09-11T14:31:56.116 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:31:56.116 INFO:teuthology.orchestra.run.smithi181.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-11T14:31:56.116 INFO:teuthology.orchestra.run.smithi181.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-11T14:31:56.116 INFO:teuthology.orchestra.run.smithi181.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-11T14:31:56.116 INFO:teuthology.orchestra.run.smithi181.stdout:hardware devices. 2024-09-11T14:31:56.116 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:31:56.117 INFO:teuthology.orchestra.run.smithi181.stdout:This subpackage provides tools for support of all platforms. 2024-09-11T14:31:56.117 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-11T14:31:56.117 INFO:teuthology.task.kernel:Updating grub on smithi181 to boot 5.14.0-505.el9.x86_64 2024-09-11T14:31:56.118 DEBUG:teuthology.orchestra.run.smithi181:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-11T14:31:56.770 INFO:teuthology.orchestra.run.smithi181.stderr:Generating grub configuration file ... 2024-09-11T14:31:58.101 INFO:teuthology.orchestra.run.smithi181.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-11T14:31:58.136 INFO:teuthology.orchestra.run.smithi181.stderr:done 2024-09-11T14:31:58.139 DEBUG:teuthology.orchestra.run.smithi181:> mktemp 2024-09-11T14:31:58.154 INFO:teuthology.orchestra.run.smithi181.stdout:/tmp/tmp.tksYzFEPEq 2024-09-11T14:31:58.155 DEBUG:teuthology.orchestra.run.smithi181:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.tksYzFEPEq 2024-09-11T14:31:58.221 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0666 /tmp/tmp.tksYzFEPEq 2024-09-11T14:31:58.357 DEBUG:teuthology.orchestra.remote:smithi181:/tmp/tmp.tksYzFEPEq is 6KB 2024-09-11T14:31:58.405 DEBUG:teuthology.orchestra.run.smithi181:> rm -fr /tmp/tmp.tksYzFEPEq 2024-09-11T14:31:58.420 DEBUG:teuthology.orchestra.run.smithi181:> sudo /bin/ls /boot/loader/entries || true 2024-09-11T14:31:58.485 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-11T14:31:58.486 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-11T14:31:58.486 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-11T14:31:58.486 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-11T14:31:58.487 DEBUG:teuthology.orchestra.run.smithi181:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-11T14:31:58.645 DEBUG:teuthology.orchestra.run.smithi181:> sudo shutdown -r now 2024-09-11T14:32:16.862 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:32:16.906 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:32:17.259 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:32:17.260 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-11T14:32:17.260 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T14:32:17.260 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T14:32:18.092 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:32:18.092 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:32:18.092 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-11T14:32:18.092 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-11T14:32:18.092 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-11T14:32:18.092 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-11T14:32:18.092 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-11T14:32:18.092 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:32:18.092 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-11T14:32:18.432 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T14:32:18.534 INFO:teuthology.orchestra.run.smithi203.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T14:32:18.534 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T14:32:18.534 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-11T14:32:18.534 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T14:32:18.534 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T14:32:18.535 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-09-11T14:32:18.535 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi203.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T14:32:18.535 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T14:32:18.595 INFO:teuthology.orchestra.run.smithi203.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T14:32:18.596 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T14:32:18.596 DEBUG:teuthology.orchestra.run.smithi203:> sudo rpm -qi grub2-tools 2024-09-11T14:32:18.631 INFO:teuthology.orchestra.run.smithi203.stdout:Name : grub2-tools 2024-09-11T14:32:18.631 INFO:teuthology.orchestra.run.smithi203.stdout:Epoch : 1 2024-09-11T14:32:18.632 INFO:teuthology.orchestra.run.smithi203.stdout:Version : 2.06 2024-09-11T14:32:18.632 INFO:teuthology.orchestra.run.smithi203.stdout:Release : 82.el9 2024-09-11T14:32:18.632 INFO:teuthology.orchestra.run.smithi203.stdout:Architecture: x86_64 2024-09-11T14:32:18.632 INFO:teuthology.orchestra.run.smithi203.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-11T14:32:18.632 INFO:teuthology.orchestra.run.smithi203.stdout:Group : Unspecified 2024-09-11T14:32:18.632 INFO:teuthology.orchestra.run.smithi203.stdout:Size : 8274670 2024-09-11T14:32:18.632 INFO:teuthology.orchestra.run.smithi203.stdout:License : GPLv3+ 2024-09-11T14:32:18.632 INFO:teuthology.orchestra.run.smithi203.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-11T14:32:18.633 INFO:teuthology.orchestra.run.smithi203.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-11T14:32:18.633 INFO:teuthology.orchestra.run.smithi203.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-11T14:32:18.633 INFO:teuthology.orchestra.run.smithi203.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-11T14:32:18.633 INFO:teuthology.orchestra.run.smithi203.stdout:Packager : builder@centos.org 2024-09-11T14:32:18.633 INFO:teuthology.orchestra.run.smithi203.stdout:Vendor : CentOS 2024-09-11T14:32:18.633 INFO:teuthology.orchestra.run.smithi203.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-11T14:32:18.633 INFO:teuthology.orchestra.run.smithi203.stdout:Summary : Support tools for GRUB. 2024-09-11T14:32:18.634 INFO:teuthology.orchestra.run.smithi203.stdout:Description : 2024-09-11T14:32:18.634 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:32:18.634 INFO:teuthology.orchestra.run.smithi203.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-11T14:32:18.634 INFO:teuthology.orchestra.run.smithi203.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-11T14:32:18.634 INFO:teuthology.orchestra.run.smithi203.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-11T14:32:18.634 INFO:teuthology.orchestra.run.smithi203.stdout:hardware devices. 2024-09-11T14:32:18.634 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:32:18.635 INFO:teuthology.orchestra.run.smithi203.stdout:This subpackage provides tools for support of all platforms. 2024-09-11T14:32:18.636 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-11T14:32:18.636 INFO:teuthology.task.kernel:Updating grub on smithi203 to boot 5.14.0-505.el9.x86_64 2024-09-11T14:32:18.636 DEBUG:teuthology.orchestra.run.smithi203:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-11T14:32:19.293 INFO:teuthology.orchestra.run.smithi203.stderr:Generating grub configuration file ... 2024-09-11T14:32:20.633 INFO:teuthology.orchestra.run.smithi203.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-11T14:32:20.660 INFO:teuthology.orchestra.run.smithi203.stderr:done 2024-09-11T14:32:20.662 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-09-11T14:32:20.678 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.LRL3gEWAJN 2024-09-11T14:32:20.678 DEBUG:teuthology.orchestra.run.smithi203:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.LRL3gEWAJN 2024-09-11T14:32:20.746 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.LRL3gEWAJN 2024-09-11T14:32:20.889 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.LRL3gEWAJN is 6KB 2024-09-11T14:32:20.938 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.LRL3gEWAJN 2024-09-11T14:32:20.953 DEBUG:teuthology.orchestra.run.smithi203:> sudo /bin/ls /boot/loader/entries || true 2024-09-11T14:32:21.015 INFO:teuthology.orchestra.run.smithi203.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-11T14:32:21.016 INFO:teuthology.orchestra.run.smithi203.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-11T14:32:21.016 INFO:teuthology.orchestra.run.smithi203.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-11T14:32:21.016 INFO:teuthology.orchestra.run.smithi203.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-11T14:32:21.017 DEBUG:teuthology.orchestra.run.smithi203:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-11T14:32:21.173 DEBUG:teuthology.orchestra.run.smithi203:> sudo shutdown -r now 2024-09-11T14:32:28.653 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-11T14:32:28.653 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-11T14:32:28.654 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T14:32:47.165 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-09-11T14:32:51.179 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-09-11T14:32:51.179 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-09-11T14:32:51.180 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T14:32:56.169 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-11T14:32:56.169 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T14:32:59.229 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-09-11T14:33:09.693 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-09-11T14:33:11.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-11T14:33:11.231 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T14:33:18.701 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-09-11T14:33:18.701 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T14:33:21.757 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-09-11T14:33:33.769 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-09-11T14:33:33.770 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T14:33:43.058 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-09-11T14:33:43.805 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-11T14:33:43.805 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-11T14:33:43.806 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-09-11T14:33:43.820 INFO:teuthology.orchestra.run.smithi181.stdout:5.14.0-505.el9.x86_64 2024-09-11T14:33:43.820 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-11T14:33:43.821 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T14:33:43.821 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-11T14:33:44.821 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-11T14:33:44.821 DEBUG:teuthology.orchestra.run.smithi181:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T14:33:44.878 INFO:teuthology.orchestra.run.smithi181.stdout:ttyS1 2024-09-11T14:33:44.902 DEBUG:teuthology.parallel:result is None 2024-09-11T14:33:52.189 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-09-11T14:34:07.190 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-09-11T14:34:07.191 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T14:34:07.621 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-09-11T14:34:08.017 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-09-11T14:34:08.017 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.14.0-505.el9.x86_64"... 2024-09-11T14:34:08.017 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2024-09-11T14:34:08.031 INFO:teuthology.orchestra.run.smithi203.stdout:5.14.0-505.el9.x86_64 2024-09-11T14:34:08.032 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-11T14:34:08.032 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T14:34:08.032 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-11T14:34:09.033 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-09-11T14:34:09.033 DEBUG:teuthology.orchestra.run.smithi203:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T14:34:09.116 INFO:teuthology.orchestra.run.smithi203.stdout:ttyS1 2024-09-11T14:34:09.145 DEBUG:teuthology.parallel:result is None 2024-09-11T14:34:09.146 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-11T14:34:09.160 INFO:teuthology.task.internal:Creating test directory... 2024-09-11T14:34:09.161 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T14:34:09.164 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T14:34:09.180 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-11T14:34:09.186 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-11T14:34:09.196 INFO:teuthology.task.internal:Creating archive directory... 2024-09-11T14:34:09.196 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T14:34:09.223 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T14:34:09.247 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-11T14:34:09.253 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-11T14:34:09.253 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T14:34:09.284 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T14:34:09.313 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T14:34:09.323 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T14:34:09.347 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T14:34:09.359 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T14:34:09.360 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-11T14:34:09.367 INFO:teuthology.task.internal:Configuring sudo... 2024-09-11T14:34:09.367 DEBUG:teuthology.orchestra.run.smithi181:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T14:34:09.369 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T14:34:09.395 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-11T14:34:09.403 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-11T14:34:09.404 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T14:34:09.437 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T14:34:09.453 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T14:34:09.540 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T14:34:09.583 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T14:34:09.660 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T14:34:09.707 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-11T14:34:09.707 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T14:34:09.772 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T14:34:09.815 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T14:34:09.865 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T14:34:09.950 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T14:34:09.994 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-11T14:34:09.995 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T14:34:10.063 DEBUG:teuthology.orchestra.run.smithi181:> sudo service rsyslog restart 2024-09-11T14:34:10.065 DEBUG:teuthology.orchestra.run.smithi203:> sudo service rsyslog restart 2024-09-11T14:34:10.125 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T14:34:10.157 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T14:34:10.638 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-11T14:34:10.649 INFO:teuthology.task.internal:Starting timer... 2024-09-11T14:34:10.649 INFO:teuthology.run_tasks:Running task pcp... 2024-09-11T14:34:10.660 INFO:teuthology.run_tasks:Running task selinux... 2024-09-11T14:34:10.669 DEBUG:teuthology.orchestra.run.smithi181:> sudo service auditd rotate 2024-09-11T14:34:10.740 INFO:teuthology.orchestra.run.smithi181.stdout:Rotating logs: 2024-09-11T14:34:10.742 DEBUG:teuthology.orchestra.run.smithi203:> sudo service auditd rotate 2024-09-11T14:34:10.810 INFO:teuthology.orchestra.run.smithi203.stdout:Rotating logs: 2024-09-11T14:34:10.812 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-11T14:34:10.812 DEBUG:teuthology.orchestra.run.smithi181:> /usr/sbin/getenforce 2024-09-11T14:34:10.852 INFO:teuthology.orchestra.run.smithi181.stdout:Permissive 2024-09-11T14:34:10.853 DEBUG:teuthology.orchestra.run.smithi203:> /usr/sbin/getenforce 2024-09-11T14:34:10.890 INFO:teuthology.orchestra.run.smithi203.stdout:Permissive 2024-09-11T14:34:10.890 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi181.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi203.front.sepia.ceph.com': 'permissive'} 2024-09-11T14:34:10.890 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-11T14:34:10.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T14:34:10.921 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-11T14:34:10.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T14:34:10.958 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-11T14:34:10.958 DEBUG:teuthology.orchestra.run.smithi181:> sudo /usr/sbin/setenforce permissive 2024-09-11T14:34:10.993 DEBUG:teuthology.orchestra.run.smithi203:> sudo /usr/sbin/setenforce permissive 2024-09-11T14:34:11.039 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-11T14:34:11.049 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-11T14:34:11.053 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-11T14:34:11.188 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-11T14:34:11.199 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-11T14:34:11.200 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi181.front.sepia.ceph.com,smithi203.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-11T14:37:50.499 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi181.front.sepia.ceph.com'), Remote(name='ubuntu@smithi203.front.sepia.ceph.com')] 2024-09-11T14:37:50.500 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-11T14:37:50.501 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T14:37:50.577 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-09-11T14:37:50.656 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-11T14:37:50.656 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-09-11T14:37:50.657 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T14:37:50.740 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-09-11T14:37:50.817 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-09-11T14:37:50.817 INFO:teuthology.run_tasks:Running task clock... 2024-09-11T14:37:50.828 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-11T14:37:50.828 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T14:37:50.828 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T14:37:50.831 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T14:37:50.831 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T14:37:50.858 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-11T14:37:50.873 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-11T14:37:50.888 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-11T14:37:50.897 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: ntpd: command not found 2024-09-11T14:37:50.903 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-11T14:37:50.908 INFO:teuthology.orchestra.run.smithi181.stdout:506 Cannot talk to daemon 2024-09-11T14:37:50.923 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-11T14:37:50.929 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: ntpd: command not found 2024-09-11T14:37:50.936 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-11T14:37:50.941 INFO:teuthology.orchestra.run.smithi203.stdout:506 Cannot talk to daemon 2024-09-11T14:37:50.955 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-11T14:37:50.969 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-11T14:37:50.994 INFO:teuthology.orchestra.run.smithi181.stderr:bash: line 1: ntpq: command not found 2024-09-11T14:37:50.997 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T14:37:50.997 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-09-11T14:37:50.997 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T14:37:50.997 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T14:37:50.998 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T14:37:50.998 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T14:37:51.029 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-09-11T14:37:51.032 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T14:37:51.033 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-09-11T14:37:51.033 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T14:37:51.033 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T14:37:51.033 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T14:37:51.033 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T14:37:51.033 INFO:teuthology.run_tasks:Running task install... 2024-09-11T14:37:51.041 DEBUG:teuthology.task.install:project ceph 2024-09-11T14:37:51.041 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a'}} 2024-09-11T14:37:51.041 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a'} 2024-09-11T14:37:51.041 INFO:teuthology.task.install:Using flavor: crimson 2024-09-11T14:37:51.046 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-11T14:37:51.046 INFO:teuthology.task.install:extra packages: [] 2024-09-11T14:37:51.047 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a', 'tag': None, 'wait_for_package': False} 2024-09-11T14:37:51.047 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=0d61375b8ab53be4c233f5b93e463edfdff9316a 2024-09-11T14:37:51.049 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a', 'tag': None, 'wait_for_package': False} 2024-09-11T14:37:51.049 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=0d61375b8ab53be4c233f5b93e463edfdff9316a 2024-09-11T14:37:51.189 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/main/0d61375b8ab53be4c233f5b93e463edfdff9316a/centos/9/flavors/crimson/ 2024-09-11T14:37:51.190 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4872.g0d61375b 2024-09-11T14:37:51.193 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/main/0d61375b8ab53be4c233f5b93e463edfdff9316a/centos/9/flavors/crimson/ 2024-09-11T14:37:51.193 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4872.g0d61375b 2024-09-11T14:37:51.408 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/main/0d61375b8ab53be4c233f5b93e463edfdff9316a/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/main/0d61375b8ab53be4c233f5b93e463edfdff9316a/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/main/0d61375b8ab53be4c233f5b93e463edfdff9316a/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-11T14:37:51.408 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-11T14:37:51.408 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-11T14:37:51.425 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/main/0d61375b8ab53be4c233f5b93e463edfdff9316a/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/main/0d61375b8ab53be4c233f5b93e463edfdff9316a/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/main/0d61375b8ab53be4c233f5b93e463edfdff9316a/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-11T14:37:51.425 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-11T14:37:51.425 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-11T14:37:51.435 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-11T14:37:51.435 DEBUG:teuthology.orchestra.run.smithi181:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0d61375b8ab53be4c233f5b93e463edfdff9316a/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-11T14:37:51.453 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-11T14:37:51.453 DEBUG:teuthology.orchestra.run.smithi203:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0d61375b8ab53be4c233f5b93e463edfdff9316a/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-11T14:37:51.502 DEBUG:teuthology.orchestra.run.smithi181:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-11T14:37:51.520 DEBUG:teuthology.orchestra.run.smithi203:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-11T14:37:51.577 DEBUG:teuthology.orchestra.run.smithi181:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-11T14:37:51.597 DEBUG:teuthology.orchestra.run.smithi203:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-11T14:37:51.641 INFO:teuthology.orchestra.run.smithi181.stdout:check_obsoletes = 1 2024-09-11T14:37:51.643 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum clean all 2024-09-11T14:37:51.661 INFO:teuthology.orchestra.run.smithi203.stdout:check_obsoletes = 1 2024-09-11T14:37:51.663 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum clean all 2024-09-11T14:37:51.871 INFO:teuthology.orchestra.run.smithi181.stdout:45 files removed 2024-09-11T14:37:51.894 INFO:teuthology.orchestra.run.smithi203.stdout:45 files removed 2024-09-11T14:37:51.898 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-radosgw 2024-09-11T14:37:51.921 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-radosgw 2024-09-11T14:37:52.713 INFO:teuthology.orchestra.run.smithi181.stdout:ceph packages for x86_64 167 kB/s | 89 kB 00:00 2024-09-11T14:37:52.760 INFO:teuthology.orchestra.run.smithi203.stdout:ceph packages for x86_64 160 kB/s | 89 kB 00:00 2024-09-11T14:37:53.030 INFO:teuthology.orchestra.run.smithi181.stdout:ceph noarch packages 89 kB/s | 24 kB 00:00 2024-09-11T14:37:53.061 INFO:teuthology.orchestra.run.smithi203.stdout:ceph noarch packages 95 kB/s | 24 kB 00:00 2024-09-11T14:37:53.314 INFO:teuthology.orchestra.run.smithi181.stdout:ceph source packages 8.5 kB/s | 2.1 kB 00:00 2024-09-11T14:37:53.337 INFO:teuthology.orchestra.run.smithi203.stdout:ceph source packages 8.8 kB/s | 2.1 kB 00:00 2024-09-11T14:37:54.053 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-09-11T14:37:55.022 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 8.2 MB 00:01 2024-09-11T14:37:55.830 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - AppStream 30 MB/s | 20 MB 00:00 2024-09-11T14:37:58.341 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - AppStream 9.1 MB/s | 20 MB 00:02 2024-09-11T14:38:00.390 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - CRB 8.4 MB/s | 6.5 MB 00:00 2024-09-11T14:38:01.849 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - Extras packages 79 kB/s | 19 kB 00:00 2024-09-11T14:38:02.620 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 35 MB/s | 23 MB 00:00 2024-09-11T14:38:03.033 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - CRB 7.6 MB/s | 6.5 MB 00:00 2024-09-11T14:38:04.775 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 19 kB 00:00 2024-09-11T14:38:05.638 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 31 MB/s | 23 MB 00:00 2024-09-11T14:38:08.101 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-11T14:38:10.519 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-11T14:38:10.536 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:38:10.536 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-09-11T14:38:10.536 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:38:10.536 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-11T14:38:10.536 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 10 M 2024-09-11T14:38:10.536 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrading: 2024-09-11T14:38:10.536 INFO:teuthology.orchestra.run.smithi203.stdout: librados2 x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 3.5 M 2024-09-11T14:38:10.536 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1 x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 3.8 M 2024-09-11T14:38:10.536 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-09-11T14:38:10.536 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-11T14:38:10.537 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 6.1 M 2024-09-11T14:38:10.537 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 20 M 2024-09-11T14:38:10.537 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 25 k 2024-09-11T14:38:10.537 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-11T14:38:10.537 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-11T14:38:10.537 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-11T14:38:10.537 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2 x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 808 k 2024-09-11T14:38:10.537 INFO:teuthology.orchestra.run.smithi203.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-11T14:38:10.537 INFO:teuthology.orchestra.run.smithi203.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-11T14:38:10.537 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-11T14:38:10.537 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-11T14:38:10.537 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1 x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 558 k 2024-09-11T14:38:10.537 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-11T14:38:10.538 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2 x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 5.4 M 2024-09-11T14:38:10.538 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-11T14:38:10.538 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-11T14:38:10.538 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-11T14:38:10.538 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 45 k 2024-09-11T14:38:10.538 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 143 k 2024-09-11T14:38:10.538 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 159 k 2024-09-11T14:38:10.538 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-11T14:38:10.538 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 316 k 2024-09-11T14:38:10.538 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 303 k 2024-09-11T14:38:10.538 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 99 k 2024-09-11T14:38:10.538 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-11T14:38:10.538 INFO:teuthology.orchestra.run.smithi203.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-11T14:38:10.539 INFO:teuthology.orchestra.run.smithi203.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-11T14:38:10.539 INFO:teuthology.orchestra.run.smithi203.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-11T14:38:10.539 INFO:teuthology.orchestra.run.smithi203.stdout:Installing weak dependencies: 2024-09-11T14:38:10.539 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-11T14:38:10.539 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:38:10.539 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-11T14:38:10.539 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:38:10.539 INFO:teuthology.orchestra.run.smithi203.stdout:Install 31 Packages 2024-09-11T14:38:10.539 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrade 2 Packages 2024-09-11T14:38:10.539 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:38:10.540 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 60 M 2024-09-11T14:38:10.540 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-11T14:38:11.035 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-11T14:38:11.311 INFO:teuthology.orchestra.run.smithi203.stdout:(1/33): ceph-base-19.3.0-4872.g0d61375b.el9.x86 10 MB/s | 6.1 MB 00:00 2024-09-11T14:38:11.369 INFO:teuthology.orchestra.run.smithi203.stdout:(2/33): ceph-selinux-19.3.0-4872.g0d61375b.el9. 432 kB/s | 25 kB 00:00 2024-09-11T14:38:11.470 INFO:teuthology.orchestra.run.smithi203.stdout:(3/33): libcephfs2-19.3.0-4872.g0d61375b.el9.x8 7.9 MB/s | 808 kB 00:00 2024-09-11T14:38:11.578 INFO:teuthology.orchestra.run.smithi203.stdout:(4/33): ceph-radosgw-19.3.0-4872.g0d61375b.el9. 12 MB/s | 10 MB 00:00 2024-09-11T14:38:11.620 INFO:teuthology.orchestra.run.smithi203.stdout:(5/33): libradosstriper1-19.3.0-4872.g0d61375b. 3.6 MB/s | 558 kB 00:00 2024-09-11T14:38:11.670 INFO:teuthology.orchestra.run.smithi203.stdout:(6/33): python3-ceph-argparse-19.3.0-4872.g0d61 902 kB/s | 45 kB 00:00 2024-09-11T14:38:11.871 INFO:teuthology.orchestra.run.smithi203.stdout:(7/33): ceph-common-19.3.0-4872.g0d61375b.el9.x 17 MB/s | 20 MB 00:01 2024-09-11T14:38:11.896 INFO:teuthology.orchestra.run.smithi203.stdout:(8/33): python3-ceph-common-19.3.0-4872.g0d6137 635 kB/s | 143 kB 00:00 2024-09-11T14:38:11.930 INFO:teuthology.orchestra.run.smithi203.stdout:(9/33): python3-cephfs-19.3.0-4872.g0d61375b.el 2.7 MB/s | 159 kB 00:00 2024-09-11T14:38:11.955 INFO:teuthology.orchestra.run.smithi203.stdout:(10/33): python3-rados-19.3.0-4872.g0d61375b.el 5.3 MB/s | 316 kB 00:00 2024-09-11T14:38:11.980 INFO:teuthology.orchestra.run.smithi203.stdout:(11/33): python3-rbd-19.3.0-4872.g0d61375b.el9. 5.9 MB/s | 303 kB 00:00 2024-09-11T14:38:12.006 INFO:teuthology.orchestra.run.smithi203.stdout:(12/33): python3-rgw-19.3.0-4872.g0d61375b.el9. 1.9 MB/s | 99 kB 00:00 2024-09-11T14:38:12.081 INFO:teuthology.orchestra.run.smithi203.stdout:(13/33): librgw2-19.3.0-4872.g0d61375b.el9.x86_ 11 MB/s | 5.4 MB 00:00 2024-09-11T14:38:12.173 INFO:teuthology.orchestra.run.smithi203.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 556 kB/s | 107 kB 00:00 2024-09-11T14:38:12.223 INFO:teuthology.orchestra.run.smithi203.stdout:(15/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 901 kB/s | 45 kB 00:00 2024-09-11T14:38:12.248 INFO:teuthology.orchestra.run.smithi203.stdout:(16/33): libnbd-1.20.2-2.el9.x86_64.rpm 703 kB/s | 171 kB 00:00 2024-09-11T14:38:12.274 INFO:teuthology.orchestra.run.smithi203.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 832 kB/s | 160 kB 00:00 2024-09-11T14:38:12.324 INFO:teuthology.orchestra.run.smithi203.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 838 kB/s | 42 kB 00:00 2024-09-11T14:38:12.358 INFO:teuthology.orchestra.run.smithi203.stdout:(19/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-09-11T14:38:12.400 INFO:teuthology.orchestra.run.smithi203.stdout:(20/33): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 662 kB 00:00 2024-09-11T14:38:12.425 INFO:teuthology.orchestra.run.smithi203.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 535 kB/s | 36 kB 00:00 2024-09-11T14:38:12.451 INFO:teuthology.orchestra.run.smithi203.stdout:(22/33): qatlib-24.02.0-1.el9.x86_64.rpm 1.7 MB/s | 221 kB 00:00 2024-09-11T14:38:12.476 INFO:teuthology.orchestra.run.smithi203.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 626 kB/s | 47 kB 00:00 2024-09-11T14:38:12.526 INFO:teuthology.orchestra.run.smithi203.stdout:(24/33): gperftools-libs-2.9.1-2.el9.x86_64.rpm 3.0 MB/s | 309 kB 00:00 2024-09-11T14:38:12.551 INFO:teuthology.orchestra.run.smithi203.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 2.0 MB/s | 49 kB 00:00 2024-09-11T14:38:12.577 INFO:teuthology.orchestra.run.smithi203.stdout:(26/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 298 kB/s | 26 kB 00:00 2024-09-11T14:38:12.602 INFO:teuthology.orchestra.run.smithi203.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-11T14:38:12.636 INFO:teuthology.orchestra.run.smithi203.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-11T14:38:12.678 INFO:teuthology.orchestra.run.smithi203.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.1 MB/s | 839 kB 00:00 2024-09-11T14:38:12.737 INFO:teuthology.orchestra.run.smithi203.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-11T14:38:12.837 INFO:teuthology.orchestra.run.smithi203.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-11T14:38:12.921 INFO:teuthology.orchestra.run.smithi203.stdout:(32/33): librados2-19.3.0-4872.g0d61375b.el9.x8 14 MB/s | 3.5 MB 00:00 2024-09-11T14:38:13.020 INFO:teuthology.orchestra.run.smithi203.stdout:(33/33): librbd1-19.3.0-4872.g0d61375b.el9.x86_ 13 MB/s | 3.8 MB 00:00 2024-09-11T14:38:13.025 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:38:13.025 INFO:teuthology.orchestra.run.smithi203.stdout:Total 24 MB/s | 60 MB 00:02 2024-09-11T14:38:13.136 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-11T14:38:13.173 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-11T14:38:13.174 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-11T14:38:13.361 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T14:38:13.378 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:38:13.378 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-11T14:38:13.379 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:38:13.379 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T14:38:13.379 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 10 M 2024-09-11T14:38:13.379 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrading: 2024-09-11T14:38:13.379 INFO:teuthology.orchestra.run.smithi181.stdout: librados2 x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 3.5 M 2024-09-11T14:38:13.379 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1 x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 3.8 M 2024-09-11T14:38:13.379 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-11T14:38:13.379 INFO:teuthology.orchestra.run.smithi181.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-11T14:38:13.379 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 6.1 M 2024-09-11T14:38:13.379 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 20 M 2024-09-11T14:38:13.379 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 25 k 2024-09-11T14:38:13.379 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-11T14:38:13.379 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-11T14:38:13.379 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-11T14:38:13.380 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2 x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 808 k 2024-09-11T14:38:13.380 INFO:teuthology.orchestra.run.smithi181.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-11T14:38:13.380 INFO:teuthology.orchestra.run.smithi181.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-11T14:38:13.380 INFO:teuthology.orchestra.run.smithi181.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-11T14:38:13.380 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-11T14:38:13.380 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1 x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 558 k 2024-09-11T14:38:13.380 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-11T14:38:13.380 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2 x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 5.4 M 2024-09-11T14:38:13.380 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-11T14:38:13.380 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-11T14:38:13.380 INFO:teuthology.orchestra.run.smithi181.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-11T14:38:13.380 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 45 k 2024-09-11T14:38:13.380 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 143 k 2024-09-11T14:38:13.381 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 159 k 2024-09-11T14:38:13.381 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-11T14:38:13.381 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 316 k 2024-09-11T14:38:13.381 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 303 k 2024-09-11T14:38:13.381 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 99 k 2024-09-11T14:38:13.381 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-11T14:38:13.381 INFO:teuthology.orchestra.run.smithi181.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-11T14:38:13.381 INFO:teuthology.orchestra.run.smithi181.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-11T14:38:13.381 INFO:teuthology.orchestra.run.smithi181.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-11T14:38:13.381 INFO:teuthology.orchestra.run.smithi181.stdout:Installing weak dependencies: 2024-09-11T14:38:13.381 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-11T14:38:13.381 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:38:13.382 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T14:38:13.382 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:38:13.382 INFO:teuthology.orchestra.run.smithi181.stdout:Install 31 Packages 2024-09-11T14:38:13.382 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrade 2 Packages 2024-09-11T14:38:13.382 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:38:13.382 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 60 M 2024-09-11T14:38:13.382 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T14:38:13.862 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-11T14:38:13.862 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-11T14:38:14.253 INFO:teuthology.orchestra.run.smithi181.stdout:(1/33): ceph-base-19.3.0-4872.g0d61375b.el9.x86 11 MB/s | 6.1 MB 00:00 2024-09-11T14:38:14.303 INFO:teuthology.orchestra.run.smithi181.stdout:(2/33): ceph-selinux-19.3.0-4872.g0d61375b.el9. 502 kB/s | 25 kB 00:00 2024-09-11T14:38:14.403 INFO:teuthology.orchestra.run.smithi181.stdout:(3/33): libcephfs2-19.3.0-4872.g0d61375b.el9.x8 7.9 MB/s | 808 kB 00:00 2024-09-11T14:38:14.562 INFO:teuthology.orchestra.run.smithi181.stdout:(4/33): ceph-radosgw-19.3.0-4872.g0d61375b.el9. 12 MB/s | 10 MB 00:00 2024-09-11T14:38:14.596 INFO:teuthology.orchestra.run.smithi181.stdout:(5/33): libradosstriper1-19.3.0-4872.g0d61375b. 2.8 MB/s | 558 kB 00:00 2024-09-11T14:38:14.788 INFO:teuthology.orchestra.run.smithi181.stdout:(6/33): ceph-common-19.3.0-4872.g0d61375b.el9.x 18 MB/s | 20 MB 00:01 2024-09-11T14:38:14.813 INFO:teuthology.orchestra.run.smithi181.stdout:(7/33): python3-ceph-argparse-19.3.0-4872.g0d61 208 kB/s | 45 kB 00:00 2024-09-11T14:38:14.838 INFO:teuthology.orchestra.run.smithi181.stdout:(8/33): python3-ceph-common-19.3.0-4872.g0d6137 2.8 MB/s | 143 kB 00:00 2024-09-11T14:38:14.863 INFO:teuthology.orchestra.run.smithi181.stdout:(9/33): python3-cephfs-19.3.0-4872.g0d61375b.el 3.1 MB/s | 159 kB 00:00 2024-09-11T14:38:14.889 INFO:teuthology.orchestra.run.smithi181.stdout:(10/33): python3-rados-19.3.0-4872.g0d61375b.el 6.1 MB/s | 316 kB 00:00 2024-09-11T14:38:14.914 INFO:teuthology.orchestra.run.smithi181.stdout:(11/33): python3-rbd-19.3.0-4872.g0d61375b.el9. 5.9 MB/s | 303 kB 00:00 2024-09-11T14:38:14.939 INFO:teuthology.orchestra.run.smithi181.stdout:(12/33): python3-rgw-19.3.0-4872.g0d61375b.el9. 1.9 MB/s | 99 kB 00:00 2024-09-11T14:38:15.015 INFO:teuthology.orchestra.run.smithi181.stdout:(13/33): librgw2-19.3.0-4872.g0d61375b.el9.x86_ 12 MB/s | 5.4 MB 00:00 2024-09-11T14:38:15.115 INFO:teuthology.orchestra.run.smithi181.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 533 kB/s | 107 kB 00:00 2024-09-11T14:38:15.140 INFO:teuthology.orchestra.run.smithi181.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 851 kB/s | 171 kB 00:00 2024-09-11T14:38:15.165 INFO:teuthology.orchestra.run.smithi181.stdout:(16/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-09-11T14:38:15.191 INFO:teuthology.orchestra.run.smithi181.stdout:(17/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 601 kB/s | 45 kB 00:00 2024-09-11T14:38:15.232 INFO:teuthology.orchestra.run.smithi181.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-11T14:38:15.258 INFO:teuthology.orchestra.run.smithi181.stdout:(19/33): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-09-11T14:38:15.291 INFO:teuthology.orchestra.run.smithi181.stdout:(20/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-11T14:38:15.316 INFO:teuthology.orchestra.run.smithi181.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 614 kB/s | 36 kB 00:00 2024-09-11T14:38:15.342 INFO:teuthology.orchestra.run.smithi181.stdout:(22/33): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-09-11T14:38:15.367 INFO:teuthology.orchestra.run.smithi181.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 628 kB/s | 47 kB 00:00 2024-09-11T14:38:15.395 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-11T14:38:15.425 INFO:teuthology.orchestra.run.smithi181.stdout:(24/33): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.8 MB/s | 309 kB 00:00 2024-09-11T14:38:15.443 INFO:teuthology.orchestra.run.smithi181.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 3.2 MB/s | 49 kB 00:00 2024-09-11T14:38:15.459 INFO:teuthology.orchestra.run.smithi181.stdout:(26/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 337 kB/s | 26 kB 00:00 2024-09-11T14:38:15.477 INFO:teuthology.orchestra.run.smithi181.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-09-11T14:38:15.510 INFO:teuthology.orchestra.run.smithi181.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.7 MB/s | 191 kB 00:00 2024-09-11T14:38:15.552 INFO:teuthology.orchestra.run.smithi181.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.9 MB/s | 839 kB 00:00 2024-09-11T14:38:15.602 INFO:teuthology.orchestra.run.smithi181.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-09-11T14:38:15.669 INFO:teuthology.orchestra.run.smithi181.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-09-11T14:38:15.778 INFO:teuthology.orchestra.run.smithi181.stdout:(32/33): librados2-19.3.0-4872.g0d61375b.el9.x8 15 MB/s | 3.5 MB 00:00 2024-09-11T14:38:15.878 INFO:teuthology.orchestra.run.smithi181.stdout:(33/33): librbd1-19.3.0-4872.g0d61375b.el9.x86_ 14 MB/s | 3.8 MB 00:00 2024-09-11T14:38:15.883 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:38:15.883 INFO:teuthology.orchestra.run.smithi181.stdout:Total 24 MB/s | 60 MB 00:02 2024-09-11T14:38:15.929 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-11T14:38:15.991 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T14:38:15.998 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librados2-2:19.3.0-4872.g0d61375b.el9.x86_64 2/35 2024-09-11T14:38:16.028 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T14:38:16.028 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T14:38:16.089 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:19.3.0-4872.g0d61375b.el9.x86_64 2/35 2024-09-11T14:38:16.179 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rados-2:19.3.0-4872.g0d61375b.el9.x86_64 3/35 2024-09-11T14:38:16.208 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs2-2:19.3.0-4872.g0d61375b.el9.x86_64 4/35 2024-09-11T14:38:16.271 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs2-2:19.3.0-4872.g0d61375b.el9.x86_64 4/35 2024-09-11T14:38:16.324 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/35 2024-09-11T14:38:16.355 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/35 2024-09-11T14:38:16.380 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/35 2024-09-11T14:38:16.403 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-11T14:38:16.429 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-11T14:38:16.628 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-11T14:38:16.661 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-11T14:38:16.696 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/35 2024-09-11T14:38:16.703 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T14:38:16.704 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T14:38:16.740 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-argparse-2:19.3.0-4872.g0d61375b.el9. 11/35 2024-09-11T14:38:16.769 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cephfs-2:19.3.0-4872.g0d61375b.el9.x86_64 12/35 2024-09-11T14:38:16.849 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 13/35 2024-09-11T14:38:16.883 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libradosstriper1-2:19.3.0-4872.g0d61375b.el9.x86_6 14/35 2024-09-11T14:38:16.972 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4872.g0d61375b.el9.x86_6 14/35 2024-09-11T14:38:17.007 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : thrift-0.15.0-2.el9.x86_64 15/35 2024-09-11T14:38:17.119 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : re2-1:20211101-20.el9.x86_64 16/35 2024-09-11T14:38:17.150 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 17/35 2024-09-11T14:38:17.208 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 18/35 2024-09-11T14:38:17.258 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 19/35 2024-09-11T14:38:17.287 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 20/35 2024-09-11T14:38:17.324 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 21/35 2024-09-11T14:38:17.684 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-11T14:38:17.741 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librbd1-2:19.3.0-4872.g0d61375b.el9.x86_64 23/35 2024-09-11T14:38:17.832 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:19.3.0-4872.g0d61375b.el9.x86_64 23/35 2024-09-11T14:38:17.863 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rbd-2:19.3.0-4872.g0d61375b.el9.x86_64 24/35 2024-09-11T14:38:17.898 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-11T14:38:18.208 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T14:38:18.482 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-11T14:38:18.511 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librgw2-2:19.3.0-4872.g0d61375b.el9.x86_64 27/35 2024-09-11T14:38:18.582 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librgw2-2:19.3.0-4872.g0d61375b.el9.x86_64 27/35 2024-09-11T14:38:18.651 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rgw-2:19.3.0-4872.g0d61375b.el9.x86_64 28/35 2024-09-11T14:38:18.680 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-common-2:19.3.0-4872.g0d61375b.el9.x8 29/35 2024-09-11T14:38:18.740 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-11T14:38:18.789 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librados2-2:19.3.0-4872.g0d61375b.el9.x86_64 2/35 2024-09-11T14:38:18.863 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:19.3.0-4872.g0d61375b.el9.x86_64 2/35 2024-09-11T14:38:18.952 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rados-2:19.3.0-4872.g0d61375b.el9.x86_64 3/35 2024-09-11T14:38:18.982 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs2-2:19.3.0-4872.g0d61375b.el9.x86_64 4/35 2024-09-11T14:38:19.044 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs2-2:19.3.0-4872.g0d61375b.el9.x86_64 4/35 2024-09-11T14:38:19.082 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/35 2024-09-11T14:38:19.111 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/35 2024-09-11T14:38:19.137 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/35 2024-09-11T14:38:19.159 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-11T14:38:19.178 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-11T14:38:19.374 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-11T14:38:19.409 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-11T14:38:19.443 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/35 2024-09-11T14:38:19.487 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-argparse-2:19.3.0-4872.g0d61375b.el9. 11/35 2024-09-11T14:38:19.517 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cephfs-2:19.3.0-4872.g0d61375b.el9.x86_64 12/35 2024-09-11T14:38:19.606 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 13/35 2024-09-11T14:38:19.639 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libradosstriper1-2:19.3.0-4872.g0d61375b.el9.x86_6 14/35 2024-09-11T14:38:19.728 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4872.g0d61375b.el9.x86_6 14/35 2024-09-11T14:38:19.771 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : thrift-0.15.0-2.el9.x86_64 15/35 2024-09-11T14:38:19.882 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : re2-1:20211101-20.el9.x86_64 16/35 2024-09-11T14:38:19.914 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 17/35 2024-09-11T14:38:19.955 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 18/35 2024-09-11T14:38:19.995 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 19/35 2024-09-11T14:38:20.024 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 20/35 2024-09-11T14:38:20.072 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 21/35 2024-09-11T14:38:20.446 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-11T14:38:20.503 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librbd1-2:19.3.0-4872.g0d61375b.el9.x86_64 23/35 2024-09-11T14:38:20.594 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:19.3.0-4872.g0d61375b.el9.x86_64 23/35 2024-09-11T14:38:20.625 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rbd-2:19.3.0-4872.g0d61375b.el9.x86_64 24/35 2024-09-11T14:38:20.661 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-11T14:38:21.167 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 30/35 2024-09-11T14:38:21.202 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 30/35 2024-09-11T14:38:21.239 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-11T14:38:21.264 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librgw2-2:19.3.0-4872.g0d61375b.el9.x86_64 27/35 2024-09-11T14:38:21.335 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librgw2-2:19.3.0-4872.g0d61375b.el9.x86_64 27/35 2024-09-11T14:38:21.404 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rgw-2:19.3.0-4872.g0d61375b.el9.x86_64 28/35 2024-09-11T14:38:21.441 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-common-2:19.3.0-4872.g0d61375b.el9.x8 29/35 2024-09-11T14:38:21.904 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 30/35 2024-09-11T14:38:21.948 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-base-2:19.3.0-4872.g0d61375b.el9.x86_64 31/35 2024-09-11T14:38:22.040 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-base-2:19.3.0-4872.g0d61375b.el9.x86_64 31/35 2024-09-11T14:38:22.041 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-11T14:38:22.041 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-11T14:38:22.041 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:38:22.091 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-selinux-2:19.3.0-4872.g0d61375b.el9.x86_64 32/35 2024-09-11T14:38:23.897 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 30/35 2024-09-11T14:38:23.938 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 30/35 2024-09-11T14:38:24.631 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 30/35 2024-09-11T14:38:24.666 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-base-2:19.3.0-4872.g0d61375b.el9.x86_64 31/35 2024-09-11T14:38:24.758 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-base-2:19.3.0-4872.g0d61375b.el9.x86_64 31/35 2024-09-11T14:38:24.758 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-11T14:38:24.758 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-11T14:38:24.758 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:38:24.793 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-selinux-2:19.3.0-4872.g0d61375b.el9.x86_64 32/35 2024-09-11T14:38:32.413 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4872.g0d61375b.el9.x86_64 32/35 2024-09-11T14:38:32.414 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /sys 2024-09-11T14:38:32.414 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /proc 2024-09-11T14:38:32.414 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /mnt 2024-09-11T14:38:32.414 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /var/tmp 2024-09-11T14:38:32.414 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /home 2024-09-11T14:38:32.414 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /root 2024-09-11T14:38:32.414 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /tmp 2024-09-11T14:38:32.414 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:38:33.526 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-radosgw-2:19.3.0-4872.g0d61375b.el9.x86_64 33/35 2024-09-11T14:38:33.552 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4872.g0d61375b.el9.x86_64 33/35 2024-09-11T14:38:33.552 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:38:33.552 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-11T14:38:33.552 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T14:38:33.552 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T14:38:33.552 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:38:33.554 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-11T14:38:33.637 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-11T14:38:33.637 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-11T14:38:34.957 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-11T14:38:34.957 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-base-2:19.3.0-4872.g0d61375b.el9.x86_64 1/35 2024-09-11T14:38:34.957 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 2/35 2024-09-11T14:38:34.958 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-radosgw-2:19.3.0-4872.g0d61375b.el9.x86_64 3/35 2024-09-11T14:38:34.958 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-selinux-2:19.3.0-4872.g0d61375b.el9.x86_64 4/35 2024-09-11T14:38:34.958 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs2-2:19.3.0-4872.g0d61375b.el9.x86_64 5/35 2024-09-11T14:38:34.958 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libradosstriper1-2:19.3.0-4872.g0d61375b.el9.x86_6 6/35 2024-09-11T14:38:34.958 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librgw2-2:19.3.0-4872.g0d61375b.el9.x86_64 7/35 2024-09-11T14:38:34.958 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4872.g0d61375b.el9. 8/35 2024-09-11T14:38:34.958 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-common-2:19.3.0-4872.g0d61375b.el9.x8 9/35 2024-09-11T14:38:34.958 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cephfs-2:19.3.0-4872.g0d61375b.el9.x86_64 10/35 2024-09-11T14:38:34.958 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rados-2:19.3.0-4872.g0d61375b.el9.x86_64 11/35 2024-09-11T14:38:34.958 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rbd-2:19.3.0-4872.g0d61375b.el9.x86_64 12/35 2024-09-11T14:38:34.958 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rgw-2:19.3.0-4872.g0d61375b.el9.x86_64 13/35 2024-09-11T14:38:34.958 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-11T14:38:34.958 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-11T14:38:34.959 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-11T14:38:34.959 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-11T14:38:34.959 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-11T14:38:34.959 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-11T14:38:34.959 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-11T14:38:34.959 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-11T14:38:34.959 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-11T14:38:34.959 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-11T14:38:34.959 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 24/35 2024-09-11T14:38:34.959 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-11T14:38:34.959 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-11T14:38:34.959 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-11T14:38:34.960 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-11T14:38:34.960 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-11T14:38:34.961 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-11T14:38:34.961 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-11T14:38:34.961 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:19.3.0-4872.g0d61375b.el9.x86_64 32/35 2024-09-11T14:38:34.961 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-11T14:38:34.961 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:19.3.0-4872.g0d61375b.el9.x86_64 34/35 2024-09-11T14:38:35.299 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4872.g0d61375b.el9.x86_64 32/35 2024-09-11T14:38:35.299 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /sys 2024-09-11T14:38:35.299 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /proc 2024-09-11T14:38:35.299 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /mnt 2024-09-11T14:38:35.300 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /var/tmp 2024-09-11T14:38:35.300 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /home 2024-09-11T14:38:35.300 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /root 2024-09-11T14:38:35.300 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /tmp 2024-09-11T14:38:35.300 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:38:35.833 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-11T14:38:35.834 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:38:35.834 INFO:teuthology.orchestra.run.smithi203.stdout:Upgraded: 2024-09-11T14:38:35.834 INFO:teuthology.orchestra.run.smithi203.stdout: librados2-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:35.834 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:35.834 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-11T14:38:35.834 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-11T14:38:35.834 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:35.834 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:35.834 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:35.835 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:35.835 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-11T14:38:35.835 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-11T14:38:35.835 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-11T14:38:35.835 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:35.835 INFO:teuthology.orchestra.run.smithi203.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-11T14:38:35.835 INFO:teuthology.orchestra.run.smithi203.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-11T14:38:35.835 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-11T14:38:35.835 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-11T14:38:35.835 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:35.835 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-11T14:38:35.835 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:35.836 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-11T14:38:35.836 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-11T14:38:35.836 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-11T14:38:35.836 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:35.836 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:35.836 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:35.836 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-11T14:38:35.836 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:35.836 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:35.836 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:35.837 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-11T14:38:35.837 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-11T14:38:35.837 INFO:teuthology.orchestra.run.smithi203.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-11T14:38:35.838 INFO:teuthology.orchestra.run.smithi203.stdout: re2-1:20211101-20.el9.x86_64 2024-09-11T14:38:35.838 INFO:teuthology.orchestra.run.smithi203.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-11T14:38:35.838 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:38:35.838 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-11T14:38:36.202 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-test 2024-09-11T14:38:36.419 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-radosgw-2:19.3.0-4872.g0d61375b.el9.x86_64 33/35 2024-09-11T14:38:36.444 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4872.g0d61375b.el9.x86_64 33/35 2024-09-11T14:38:36.445 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:38:36.445 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-11T14:38:36.445 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T14:38:36.445 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T14:38:36.445 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:38:36.447 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-11T14:38:36.538 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-11T14:38:36.539 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-11T14:38:36.770 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:28 ago on Wed 11 Sep 2024 02:38:08 PM UTC. 2024-09-11T14:38:36.903 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-11T14:38:36.904 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:38:36.904 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-09-11T14:38:36.904 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:38:36.904 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-11T14:38:36.904 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 85 M 2024-09-11T14:38:36.904 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-09-11T14:38:36.904 INFO:teuthology.orchestra.run.smithi203.stdout: libasan8 x86_64 14.2.1-1.2.el9 appstream 513 k 2024-09-11T14:38:36.904 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 175 k 2024-09-11T14:38:36.905 INFO:teuthology.orchestra.run.smithi203.stdout: libubsan x86_64 11.5.0-2.el9 appstream 183 k 2024-09-11T14:38:36.905 INFO:teuthology.orchestra.run.smithi203.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-09-11T14:38:36.905 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-11T14:38:36.905 INFO:teuthology.orchestra.run.smithi203.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-11T14:38:36.905 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-11T14:38:36.905 INFO:teuthology.orchestra.run.smithi203.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-09-11T14:38:36.905 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:38:36.905 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-11T14:38:36.905 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:38:36.905 INFO:teuthology.orchestra.run.smithi203.stdout:Install 9 Packages 2024-09-11T14:38:36.905 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:38:36.906 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 87 M 2024-09-11T14:38:36.906 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 797 M 2024-09-11T14:38:36.906 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-11T14:38:37.387 INFO:teuthology.orchestra.run.smithi203.stdout:(1/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 645 kB/s | 94 kB 00:00 2024-09-11T14:38:37.579 INFO:teuthology.orchestra.run.smithi203.stdout:(2/9): libasan8-14.2.1-1.2.el9.x86_64.rpm 2.6 MB/s | 513 kB 00:00 2024-09-11T14:38:37.629 INFO:teuthology.orchestra.run.smithi203.stdout:(3/9): libubsan-11.5.0-2.el9.x86_64.rpm 3.6 MB/s | 183 kB 00:00 2024-09-11T14:38:37.704 INFO:teuthology.orchestra.run.smithi203.stdout:(4/9): libcephsqlite-19.3.0-4872.g0d61375b.el9. 377 kB/s | 175 kB 00:00 2024-09-11T14:38:37.738 INFO:teuthology.orchestra.run.smithi203.stdout:(5/9): protobuf-3.14.0-13.el9.x86_64.rpm 9.3 MB/s | 1.0 MB 00:00 2024-09-11T14:38:37.780 INFO:teuthology.orchestra.run.smithi203.stdout:(6/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-11T14:38:37.822 INFO:teuthology.orchestra.run.smithi203.stdout:(7/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 2.9 MB/s | 122 kB 00:00 2024-09-11T14:38:37.901 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-11T14:38:37.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-base-2:19.3.0-4872.g0d61375b.el9.x86_64 1/35 2024-09-11T14:38:37.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 2/35 2024-09-11T14:38:37.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-radosgw-2:19.3.0-4872.g0d61375b.el9.x86_64 3/35 2024-09-11T14:38:37.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-selinux-2:19.3.0-4872.g0d61375b.el9.x86_64 4/35 2024-09-11T14:38:37.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs2-2:19.3.0-4872.g0d61375b.el9.x86_64 5/35 2024-09-11T14:38:37.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libradosstriper1-2:19.3.0-4872.g0d61375b.el9.x86_6 6/35 2024-09-11T14:38:37.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librgw2-2:19.3.0-4872.g0d61375b.el9.x86_64 7/35 2024-09-11T14:38:37.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4872.g0d61375b.el9. 8/35 2024-09-11T14:38:37.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-common-2:19.3.0-4872.g0d61375b.el9.x8 9/35 2024-09-11T14:38:37.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cephfs-2:19.3.0-4872.g0d61375b.el9.x86_64 10/35 2024-09-11T14:38:37.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rados-2:19.3.0-4872.g0d61375b.el9.x86_64 11/35 2024-09-11T14:38:37.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rbd-2:19.3.0-4872.g0d61375b.el9.x86_64 12/35 2024-09-11T14:38:37.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rgw-2:19.3.0-4872.g0d61375b.el9.x86_64 13/35 2024-09-11T14:38:37.903 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-11T14:38:37.903 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-11T14:38:37.903 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-11T14:38:37.903 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-11T14:38:37.903 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-11T14:38:37.903 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-11T14:38:37.903 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-11T14:38:37.903 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-11T14:38:37.903 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-11T14:38:37.903 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-11T14:38:37.903 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 24/35 2024-09-11T14:38:37.903 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-11T14:38:37.903 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-11T14:38:37.904 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-11T14:38:37.904 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-11T14:38:37.904 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-11T14:38:37.904 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-11T14:38:37.904 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-11T14:38:37.904 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:19.3.0-4872.g0d61375b.el9.x86_64 32/35 2024-09-11T14:38:37.904 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-11T14:38:37.904 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:19.3.0-4872.g0d61375b.el9.x86_64 34/35 2024-09-11T14:38:37.906 INFO:teuthology.orchestra.run.smithi203.stdout:(8/9): socat-1.7.4.1-6.el9.x86_64.rpm 1.5 MB/s | 304 kB 00:00 2024-09-11T14:38:39.018 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-11T14:38:39.018 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:38:39.018 INFO:teuthology.orchestra.run.smithi181.stdout:Upgraded: 2024-09-11T14:38:39.018 INFO:teuthology.orchestra.run.smithi181.stdout: librados2-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:39.018 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:39.018 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T14:38:39.019 INFO:teuthology.orchestra.run.smithi181.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-11T14:38:39.019 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:39.019 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:39.019 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:39.019 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:39.019 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-11T14:38:39.019 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-11T14:38:39.019 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-11T14:38:39.019 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:39.019 INFO:teuthology.orchestra.run.smithi181.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-11T14:38:39.019 INFO:teuthology.orchestra.run.smithi181.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-11T14:38:39.019 INFO:teuthology.orchestra.run.smithi181.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-11T14:38:39.019 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-11T14:38:39.020 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:39.020 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-11T14:38:39.020 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:39.020 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-11T14:38:39.020 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-11T14:38:39.020 INFO:teuthology.orchestra.run.smithi181.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-11T14:38:39.020 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:39.020 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:39.020 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:39.020 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-11T14:38:39.020 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:39.020 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:39.020 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:39.021 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-11T14:38:39.021 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-11T14:38:39.021 INFO:teuthology.orchestra.run.smithi181.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-11T14:38:39.021 INFO:teuthology.orchestra.run.smithi181.stdout: re2-1:20211101-20.el9.x86_64 2024-09-11T14:38:39.021 INFO:teuthology.orchestra.run.smithi181.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-11T14:38:39.021 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:38:39.021 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T14:38:39.387 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-test 2024-09-11T14:38:39.948 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:28 ago on Wed 11 Sep 2024 02:38:11 PM UTC. 2024-09-11T14:38:40.080 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T14:38:40.081 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:38:40.081 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-11T14:38:40.082 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:38:40.082 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T14:38:40.082 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 85 M 2024-09-11T14:38:40.082 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-11T14:38:40.082 INFO:teuthology.orchestra.run.smithi181.stdout: libasan8 x86_64 14.2.1-1.2.el9 appstream 513 k 2024-09-11T14:38:40.082 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 175 k 2024-09-11T14:38:40.082 INFO:teuthology.orchestra.run.smithi181.stdout: libubsan x86_64 11.5.0-2.el9 appstream 183 k 2024-09-11T14:38:40.082 INFO:teuthology.orchestra.run.smithi181.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-09-11T14:38:40.082 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-11T14:38:40.082 INFO:teuthology.orchestra.run.smithi181.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-11T14:38:40.082 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-11T14:38:40.082 INFO:teuthology.orchestra.run.smithi181.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-09-11T14:38:40.082 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:38:40.082 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T14:38:40.083 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:38:40.083 INFO:teuthology.orchestra.run.smithi181.stdout:Install 9 Packages 2024-09-11T14:38:40.083 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:38:40.083 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 87 M 2024-09-11T14:38:40.083 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 797 M 2024-09-11T14:38:40.083 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T14:38:40.473 INFO:teuthology.orchestra.run.smithi203.stdout:(9/9): ceph-test-19.3.0-4872.g0d61375b.el9.x86_ 26 MB/s | 85 MB 00:03 2024-09-11T14:38:40.476 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:38:40.476 INFO:teuthology.orchestra.run.smithi203.stdout:Total 24 MB/s | 87 MB 00:03 2024-09-11T14:38:40.561 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-11T14:38:40.568 INFO:teuthology.orchestra.run.smithi181.stdout:(1/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 521 kB/s | 94 kB 00:00 2024-09-11T14:38:40.584 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-11T14:38:40.584 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-11T14:38:40.610 INFO:teuthology.orchestra.run.smithi181.stdout:(2/9): libcephsqlite-19.3.0-4872.g0d61375b.el9. 785 kB/s | 175 kB 00:00 2024-09-11T14:38:40.819 INFO:teuthology.orchestra.run.smithi181.stdout:(3/9): libasan8-14.2.1-1.2.el9.x86_64.rpm 2.0 MB/s | 513 kB 00:00 2024-09-11T14:38:40.844 INFO:teuthology.orchestra.run.smithi181.stdout:(4/9): libubsan-11.5.0-2.el9.x86_64.rpm 782 kB/s | 183 kB 00:00 2024-09-11T14:38:40.928 INFO:teuthology.orchestra.run.smithi181.stdout:(5/9): socat-1.7.4.1-6.el9.x86_64.rpm 3.6 MB/s | 304 kB 00:00 2024-09-11T14:38:40.978 INFO:teuthology.orchestra.run.smithi181.stdout:(6/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-11T14:38:41.011 INFO:teuthology.orchestra.run.smithi181.stdout:(7/9): protobuf-3.14.0-13.el9.x86_64.rpm 5.2 MB/s | 1.0 MB 00:00 2024-09-11T14:38:41.053 INFO:teuthology.orchestra.run.smithi181.stdout:(8/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 1.6 MB/s | 122 kB 00:00 2024-09-11T14:38:41.184 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-11T14:38:41.185 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-11T14:38:42.091 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-11T14:38:42.171 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-09-11T14:38:42.243 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-09-11T14:38:42.294 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/9 2024-09-11T14:38:42.347 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-09-11T14:38:42.385 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-11T14:38:42.410 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-11T14:38:42.463 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-11T14:38:42.511 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-09-11T14:38:42.537 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephsqlite-2:19.3.0-4872.g0d61375b.el9.x86_64 8/9 2024-09-11T14:38:43.704 INFO:teuthology.orchestra.run.smithi181.stdout:(9/9): ceph-test-19.3.0-4872.g0d61375b.el9.x86_ 26 MB/s | 85 MB 00:03 2024-09-11T14:38:43.707 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:38:43.707 INFO:teuthology.orchestra.run.smithi181.stdout:Total 24 MB/s | 87 MB 00:03 2024-09-11T14:38:43.791 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T14:38:43.813 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T14:38:43.813 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T14:38:44.413 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T14:38:44.413 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T14:38:45.317 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T14:38:45.385 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-09-11T14:38:45.431 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-09-11T14:38:45.474 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/9 2024-09-11T14:38:45.509 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-09-11T14:38:45.549 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-11T14:38:45.573 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-11T14:38:45.644 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-11T14:38:45.691 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-09-11T14:38:45.717 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephsqlite-2:19.3.0-4872.g0d61375b.el9.x86_64 8/9 2024-09-11T14:38:53.999 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4872.g0d61375b.el9.x86_64 8/9 2024-09-11T14:38:54.086 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-test-2:19.3.0-4872.g0d61375b.el9.x86_64 9/9 2024-09-11T14:38:57.013 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4872.g0d61375b.el9.x86_64 8/9 2024-09-11T14:38:57.098 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-test-2:19.3.0-4872.g0d61375b.el9.x86_64 9/9 2024-09-11T14:39:00.712 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-test-2:19.3.0-4872.g0d61375b.el9.x86_64 9/9 2024-09-11T14:39:00.712 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-test-2:19.3.0-4872.g0d61375b.el9.x86_64 1/9 2024-09-11T14:39:00.712 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephsqlite-2:19.3.0-4872.g0d61375b.el9.x86_64 2/9 2024-09-11T14:39:00.712 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-09-11T14:39:00.712 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libasan8-14.2.1-1.2.el9.x86_64 4/9 2024-09-11T14:39:00.713 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-11T14:39:00.713 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-09-11T14:39:00.713 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 7/9 2024-09-11T14:39:00.713 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-09-11T14:39:01.112 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-09-11T14:39:01.112 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:39:01.112 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-11T14:39:01.112 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:39:01.112 INFO:teuthology.orchestra.run.smithi203.stdout: libasan8-14.2.1-1.2.el9.x86_64 2024-09-11T14:39:01.112 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:39:01.113 INFO:teuthology.orchestra.run.smithi203.stdout: libubsan-11.5.0-2.el9.x86_64 2024-09-11T14:39:01.113 INFO:teuthology.orchestra.run.smithi203.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-09-11T14:39:01.113 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-11T14:39:01.113 INFO:teuthology.orchestra.run.smithi203.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-11T14:39:01.113 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-11T14:39:01.113 INFO:teuthology.orchestra.run.smithi203.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-09-11T14:39:01.113 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:39:01.113 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-11T14:39:01.466 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph 2024-09-11T14:39:02.042 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:54 ago on Wed 11 Sep 2024 02:38:08 PM UTC. 2024-09-11T14:39:02.178 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-11T14:39:02.180 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:39:02.180 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-09-11T14:39:02.180 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:39:02.181 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-11T14:39:02.181 INFO:teuthology.orchestra.run.smithi203.stdout: ceph x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 6.5 k 2024-09-11T14:39:02.181 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-09-11T14:39:02.181 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 2.2 M 2024-09-11T14:39:02.181 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 1.6 M 2024-09-11T14:39:02.181 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 291 k 2024-09-11T14:39:02.181 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 5.1 M 2024-09-11T14:39:02.181 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 71 M 2024-09-11T14:39:02.181 INFO:teuthology.orchestra.run.smithi203.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-11T14:39:02.181 INFO:teuthology.orchestra.run.smithi203.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-11T14:39:02.181 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-11T14:39:02.182 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-11T14:39:02.182 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-11T14:39:02.182 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-11T14:39:02.182 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-11T14:39:02.182 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-11T14:39:02.182 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-11T14:39:02.182 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-11T14:39:02.182 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections 2024-09-11T14:39:02.182 INFO:teuthology.orchestra.run.smithi203.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-11T14:39:02.182 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-11T14:39:02.182 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-11T14:39:02.182 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-11T14:39:02.182 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-11T14:39:02.182 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-11T14:39:02.182 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-11T14:39:02.183 INFO:teuthology.orchestra.run.smithi203.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-11T14:39:02.183 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-11T14:39:02.183 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-11T14:39:02.183 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-11T14:39:02.183 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-11T14:39:02.183 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-11T14:39:02.183 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-11T14:39:02.183 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-11T14:39:02.183 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-11T14:39:02.183 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-11T14:39:02.183 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-11T14:39:02.183 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-11T14:39:02.183 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-11T14:39:02.183 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-11T14:39:02.184 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:39:02.184 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-11T14:39:02.184 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:39:02.184 INFO:teuthology.orchestra.run.smithi203.stdout:Install 37 Packages 2024-09-11T14:39:02.184 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:39:02.184 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 84 M 2024-09-11T14:39:02.185 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 896 M 2024-09-11T14:39:02.185 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-11T14:39:03.003 INFO:teuthology.orchestra.run.smithi203.stdout:(1/37): ceph-19.3.0-4872.g0d61375b.el9.x86_64.r 48 kB/s | 6.5 kB 00:00 2024-09-11T14:39:03.178 INFO:teuthology.orchestra.run.smithi203.stdout:(2/37): ceph-mgr-19.3.0-4872.g0d61375b.el9.x86_ 5.1 MB/s | 1.6 MB 00:00 2024-09-11T14:39:03.236 INFO:teuthology.orchestra.run.smithi203.stdout:(3/37): ceph-mds-19.3.0-4872.g0d61375b.el9.x86_ 5.9 MB/s | 2.2 MB 00:00 2024-09-11T14:39:03.300 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-test-2:19.3.0-4872.g0d61375b.el9.x86_64 9/9 2024-09-11T14:39:03.300 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-test-2:19.3.0-4872.g0d61375b.el9.x86_64 1/9 2024-09-11T14:39:03.300 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephsqlite-2:19.3.0-4872.g0d61375b.el9.x86_64 2/9 2024-09-11T14:39:03.300 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-09-11T14:39:03.300 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libasan8-14.2.1-1.2.el9.x86_64 4/9 2024-09-11T14:39:03.300 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-11T14:39:03.300 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-09-11T14:39:03.300 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 7/9 2024-09-11T14:39:03.300 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-09-11T14:39:03.312 INFO:teuthology.orchestra.run.smithi203.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4872.g0d61 3.8 MB/s | 291 kB 00:00 2024-09-11T14:39:03.387 INFO:teuthology.orchestra.run.smithi203.stdout:(5/37): ceph-mon-19.3.0-4872.g0d61375b.el9.x86_ 13 MB/s | 5.1 MB 00:00 2024-09-11T14:39:03.504 INFO:teuthology.orchestra.run.smithi203.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-09-11T14:39:03.546 INFO:teuthology.orchestra.run.smithi203.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-11T14:39:03.588 INFO:teuthology.orchestra.run.smithi203.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-09-11T14:39:03.630 INFO:teuthology.orchestra.run.smithi203.stdout:(9/37): python3-cryptography-36.0.1-4.el9.x86_6 5.1 MB/s | 1.2 MB 00:00 2024-09-11T14:39:03.656 INFO:teuthology.orchestra.run.smithi203.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 1.8 MB/s | 125 kB 00:00 2024-09-11T14:39:03.694 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-09-11T14:39:03.694 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:39:03.694 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T14:39:03.694 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:39:03.694 INFO:teuthology.orchestra.run.smithi181.stdout: libasan8-14.2.1-1.2.el9.x86_64 2024-09-11T14:39:03.694 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:39:03.694 INFO:teuthology.orchestra.run.smithi181.stdout: libubsan-11.5.0-2.el9.x86_64 2024-09-11T14:39:03.694 INFO:teuthology.orchestra.run.smithi181.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-09-11T14:39:03.694 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-11T14:39:03.694 INFO:teuthology.orchestra.run.smithi181.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-11T14:39:03.694 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-11T14:39:03.694 INFO:teuthology.orchestra.run.smithi181.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-09-11T14:39:03.695 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:39:03.695 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T14:39:03.723 INFO:teuthology.orchestra.run.smithi203.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 2.3 MB/s | 215 kB 00:00 2024-09-11T14:39:03.823 INFO:teuthology.orchestra.run.smithi203.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-09-11T14:39:03.873 INFO:teuthology.orchestra.run.smithi203.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 698 kB/s | 35 kB 00:00 2024-09-11T14:39:03.907 INFO:teuthology.orchestra.run.smithi203.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 937 kB/s | 172 kB 00:00 2024-09-11T14:39:03.932 INFO:teuthology.orchestra.run.smithi203.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.3 MB/s | 77 kB 00:00 2024-09-11T14:39:03.957 INFO:teuthology.orchestra.run.smithi203.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-11T14:39:03.983 INFO:teuthology.orchestra.run.smithi203.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 826 kB/s | 42 kB 00:00 2024-09-11T14:39:04.008 INFO:teuthology.orchestra.run.smithi203.stdout:(18/37): lua-devel-5.4.4-4.el9.x86_64.rpm 442 kB/s | 22 kB 00:00 2024-09-11T14:39:04.042 INFO:teuthology.orchestra.run.smithi203.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-09-11T14:39:04.052 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph 2024-09-11T14:39:04.067 INFO:teuthology.orchestra.run.smithi203.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 742 kB/s | 43 kB 00:00 2024-09-11T14:39:04.093 INFO:teuthology.orchestra.run.smithi203.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-09-11T14:39:04.118 INFO:teuthology.orchestra.run.smithi203.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 430 kB/s | 11 kB 00:00 2024-09-11T14:39:04.143 INFO:teuthology.orchestra.run.smithi203.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 712 kB/s | 18 kB 00:00 2024-09-11T14:39:04.168 INFO:teuthology.orchestra.run.smithi203.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 929 kB/s | 23 kB 00:00 2024-09-11T14:39:04.194 INFO:teuthology.orchestra.run.smithi203.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 778 kB/s | 19 kB 00:00 2024-09-11T14:39:04.219 INFO:teuthology.orchestra.run.smithi203.stdout:(26/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.3 MB/s | 358 kB 00:00 2024-09-11T14:39:04.244 INFO:teuthology.orchestra.run.smithi203.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-11T14:39:04.269 INFO:teuthology.orchestra.run.smithi203.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 924 kB/s | 46 kB 00:00 2024-09-11T14:39:04.295 INFO:teuthology.orchestra.run.smithi203.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-11T14:39:04.320 INFO:teuthology.orchestra.run.smithi203.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 659 kB/s | 16 kB 00:00 2024-09-11T14:39:04.345 INFO:teuthology.orchestra.run.smithi203.stdout:(31/37): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-09-11T14:39:04.371 INFO:teuthology.orchestra.run.smithi203.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-11T14:39:04.396 INFO:teuthology.orchestra.run.smithi203.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-11T14:39:04.430 INFO:teuthology.orchestra.run.smithi203.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-09-11T14:39:04.455 INFO:teuthology.orchestra.run.smithi203.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 804 kB/s | 20 kB 00:00 2024-09-11T14:39:04.480 INFO:teuthology.orchestra.run.smithi203.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-09-11T14:39:04.617 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:53 ago on Wed 11 Sep 2024 02:38:11 PM UTC. 2024-09-11T14:39:04.761 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T14:39:04.763 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:39:04.763 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-11T14:39:04.763 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:39:04.763 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T14:39:04.764 INFO:teuthology.orchestra.run.smithi181.stdout: ceph x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 6.5 k 2024-09-11T14:39:04.764 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-11T14:39:04.764 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 2.2 M 2024-09-11T14:39:04.764 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 1.6 M 2024-09-11T14:39:04.764 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 291 k 2024-09-11T14:39:04.764 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 5.1 M 2024-09-11T14:39:04.764 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 71 M 2024-09-11T14:39:04.764 INFO:teuthology.orchestra.run.smithi181.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-11T14:39:04.764 INFO:teuthology.orchestra.run.smithi181.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-11T14:39:04.764 INFO:teuthology.orchestra.run.smithi181.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-11T14:39:04.764 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-11T14:39:04.764 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-11T14:39:04.765 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-11T14:39:04.765 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-11T14:39:04.765 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-11T14:39:04.765 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-11T14:39:04.765 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-11T14:39:04.765 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-collections 2024-09-11T14:39:04.765 INFO:teuthology.orchestra.run.smithi181.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-11T14:39:04.765 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-11T14:39:04.765 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-11T14:39:04.765 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-11T14:39:04.765 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-11T14:39:04.765 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-11T14:39:04.765 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-11T14:39:04.766 INFO:teuthology.orchestra.run.smithi181.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-11T14:39:04.766 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-11T14:39:04.766 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-11T14:39:04.766 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-11T14:39:04.766 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-11T14:39:04.766 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-11T14:39:04.766 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-11T14:39:04.766 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-11T14:39:04.766 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-11T14:39:04.766 INFO:teuthology.orchestra.run.smithi181.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-11T14:39:04.766 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-11T14:39:04.766 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-11T14:39:04.766 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-11T14:39:04.766 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-11T14:39:04.767 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:39:04.767 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T14:39:04.767 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:39:04.767 INFO:teuthology.orchestra.run.smithi181.stdout:Install 37 Packages 2024-09-11T14:39:04.767 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:39:04.767 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 84 M 2024-09-11T14:39:04.768 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 896 M 2024-09-11T14:39:04.768 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T14:39:05.235 INFO:teuthology.orchestra.run.smithi181.stdout:(1/37): ceph-19.3.0-4872.g0d61375b.el9.x86_64.r 56 kB/s | 6.5 kB 00:00 2024-09-11T14:39:05.410 INFO:teuthology.orchestra.run.smithi181.stdout:(2/37): ceph-mds-19.3.0-4872.g0d61375b.el9.x86_ 7.5 MB/s | 2.2 MB 00:00 2024-09-11T14:39:05.460 INFO:teuthology.orchestra.run.smithi181.stdout:(3/37): ceph-mgr-19.3.0-4872.g0d61375b.el9.x86_ 4.7 MB/s | 1.6 MB 00:00 2024-09-11T14:39:05.552 INFO:teuthology.orchestra.run.smithi181.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4872.g0d61 3.1 MB/s | 291 kB 00:00 2024-09-11T14:39:05.572 INFO:teuthology.orchestra.run.smithi203.stdout:(37/37): ceph-osd-19.3.0-4872.g0d61375b.el9.x86 29 MB/s | 71 MB 00:02 2024-09-11T14:39:05.576 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:39:05.577 INFO:teuthology.orchestra.run.smithi203.stdout:Total 25 MB/s | 84 MB 00:03 2024-09-11T14:39:05.685 INFO:teuthology.orchestra.run.smithi181.stdout:(5/37): ceph-mon-19.3.0-4872.g0d61375b.el9.x86_ 11 MB/s | 5.1 MB 00:00 2024-09-11T14:39:05.730 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-11T14:39:05.772 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-11T14:39:05.773 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-11T14:39:05.827 INFO:teuthology.orchestra.run.smithi181.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 920 kB/s | 253 kB 00:00 2024-09-11T14:39:05.877 INFO:teuthology.orchestra.run.smithi181.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-11T14:39:05.928 INFO:teuthology.orchestra.run.smithi181.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 2.7 MB/s | 135 kB 00:00 2024-09-11T14:39:05.978 INFO:teuthology.orchestra.run.smithi181.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-09-11T14:39:06.020 INFO:teuthology.orchestra.run.smithi181.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 3.7 MB/s | 1.2 MB 00:00 2024-09-11T14:39:06.045 INFO:teuthology.orchestra.run.smithi181.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-09-11T14:39:06.187 INFO:teuthology.orchestra.run.smithi181.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-09-11T14:39:06.212 INFO:teuthology.orchestra.run.smithi181.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-09-11T14:39:06.238 INFO:teuthology.orchestra.run.smithi181.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 693 kB/s | 35 kB 00:00 2024-09-11T14:39:06.263 INFO:teuthology.orchestra.run.smithi181.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-09-11T14:39:06.288 INFO:teuthology.orchestra.run.smithi181.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-11T14:39:06.314 INFO:teuthology.orchestra.run.smithi181.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 826 kB/s | 42 kB 00:00 2024-09-11T14:39:06.356 INFO:teuthology.orchestra.run.smithi181.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 3.6 MB/s | 151 kB 00:00 2024-09-11T14:39:06.381 INFO:teuthology.orchestra.run.smithi181.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.7 MB/s | 43 kB 00:00 2024-09-11T14:39:06.406 INFO:teuthology.orchestra.run.smithi181.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 189 kB/s | 22 kB 00:00 2024-09-11T14:39:06.431 INFO:teuthology.orchestra.run.smithi181.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-09-11T14:39:06.457 INFO:teuthology.orchestra.run.smithi181.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 587 kB/s | 11 kB 00:00 2024-09-11T14:39:06.482 INFO:teuthology.orchestra.run.smithi181.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 765 kB/s | 18 kB 00:00 2024-09-11T14:39:06.499 INFO:teuthology.orchestra.run.smithi181.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 1.4 MB/s | 23 kB 00:00 2024-09-11T14:39:06.516 INFO:teuthology.orchestra.run.smithi181.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 1.1 MB/s | 19 kB 00:00 2024-09-11T14:39:06.533 INFO:teuthology.orchestra.run.smithi181.stdout:(26/37): python3-jaraco-text-3.2.0-6.el9.noarch 1.2 MB/s | 20 kB 00:00 2024-09-11T14:39:06.550 INFO:teuthology.orchestra.run.smithi181.stdout:(27/37): python3-logutils-0.3.5-21.el9.noarch.r 2.7 MB/s | 46 kB 00:00 2024-09-11T14:39:06.575 INFO:teuthology.orchestra.run.smithi181.stdout:(28/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.1 MB/s | 358 kB 00:00 2024-09-11T14:39:06.589 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-11T14:39:06.589 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-11T14:39:06.600 INFO:teuthology.orchestra.run.smithi181.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-11T14:39:06.626 INFO:teuthology.orchestra.run.smithi181.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 654 kB/s | 16 kB 00:00 2024-09-11T14:39:06.651 INFO:teuthology.orchestra.run.smithi181.stdout:(31/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.5 MB/s | 90 kB 00:00 2024-09-11T14:39:06.677 INFO:teuthology.orchestra.run.smithi181.stdout:(32/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-09-11T14:39:06.702 INFO:teuthology.orchestra.run.smithi181.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 710 kB/s | 36 kB 00:00 2024-09-11T14:39:06.727 INFO:teuthology.orchestra.run.smithi181.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-09-11T14:39:06.752 INFO:teuthology.orchestra.run.smithi181.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 796 kB/s | 20 kB 00:00 2024-09-11T14:39:06.781 INFO:teuthology.orchestra.run.smithi181.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.3 MB/s | 427 kB 00:00 2024-09-11T14:39:07.814 INFO:teuthology.orchestra.run.smithi181.stdout:(37/37): ceph-osd-19.3.0-4872.g0d61375b.el9.x86 29 MB/s | 71 MB 00:02 2024-09-11T14:39:07.819 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:39:07.820 INFO:teuthology.orchestra.run.smithi181.stdout:Total 28 MB/s | 84 MB 00:03 2024-09-11T14:39:07.973 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T14:39:08.014 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T14:39:08.014 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T14:39:08.281 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-11T14:39:08.333 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-11T14:39:08.375 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-11T14:39:08.402 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-11T14:39:08.475 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-11T14:39:08.518 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-11T14:39:08.551 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-11T14:39:08.643 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-11T14:39:08.746 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-11T14:39:08.787 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-11T14:39:08.823 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T14:39:08.823 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T14:39:08.831 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-11T14:39:08.861 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-11T14:39:08.895 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-11T14:39:08.929 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-11T14:39:08.961 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-11T14:39:08.996 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-11T14:39:09.031 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-11T14:39:09.079 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-11T14:39:09.111 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-11T14:39:09.184 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-11T14:39:09.323 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-11T14:39:09.403 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-11T14:39:09.473 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-11T14:39:09.511 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-11T14:39:09.560 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-11T14:39:09.624 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-11T14:39:09.811 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-11T14:39:09.857 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-11T14:39:09.937 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-11T14:39:10.024 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-11T14:39:10.057 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-11T14:39:10.192 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-11T14:39:10.386 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4872.g0d61375b.el9. 32/37 2024-09-11T14:39:10.418 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-2:19.3.0-4872.g0d61375b.el9.x86_64 33/37 2024-09-11T14:39:10.444 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4872.g0d61375b.el9.x86_64 33/37 2024-09-11T14:39:10.444 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:39:10.445 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-11T14:39:10.445 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T14:39:10.445 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T14:39:10.445 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:39:10.470 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T14:39:10.534 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-11T14:39:10.575 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-11T14:39:10.602 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-11T14:39:10.674 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-11T14:39:10.710 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-11T14:39:10.743 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-11T14:39:10.835 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-11T14:39:10.912 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-11T14:39:10.954 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-11T14:39:11.014 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-11T14:39:11.043 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-11T14:39:11.078 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-11T14:39:11.112 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-11T14:39:11.144 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-11T14:39:11.179 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-11T14:39:11.213 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-11T14:39:11.261 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-11T14:39:11.293 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-11T14:39:11.365 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-11T14:39:11.505 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-11T14:39:11.585 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-11T14:39:11.646 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-11T14:39:11.685 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-11T14:39:11.742 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-11T14:39:11.806 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-11T14:39:11.993 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-11T14:39:12.040 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-11T14:39:12.118 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-11T14:39:12.206 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-11T14:39:12.247 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-11T14:39:12.374 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-11T14:39:12.560 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4872.g0d61375b.el9. 32/37 2024-09-11T14:39:12.583 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-2:19.3.0-4872.g0d61375b.el9.x86_64 33/37 2024-09-11T14:39:12.608 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4872.g0d61375b.el9.x86_64 33/37 2024-09-11T14:39:12.608 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:39:12.608 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-11T14:39:12.608 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T14:39:12.608 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T14:39:12.608 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:39:20.968 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-osd-2:19.3.0-4872.g0d61375b.el9.x86_64 34/37 2024-09-11T14:39:20.995 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-osd-2:19.3.0-4872.g0d61375b.el9.x86_64 34/37 2024-09-11T14:39:20.996 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:39:20.996 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-11T14:39:20.996 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T14:39:20.996 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T14:39:20.996 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:39:21.536 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mon-2:19.3.0-4872.g0d61375b.el9.x86_64 35/37 2024-09-11T14:39:21.559 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mon-2:19.3.0-4872.g0d61375b.el9.x86_64 35/37 2024-09-11T14:39:21.560 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:39:21.560 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-11T14:39:21.560 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T14:39:21.560 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T14:39:21.560 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:39:21.795 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mds-2:19.3.0-4872.g0d61375b.el9.x86_64 36/37 2024-09-11T14:39:21.819 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mds-2:19.3.0-4872.g0d61375b.el9.x86_64 36/37 2024-09-11T14:39:21.819 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:39:21.819 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-11T14:39:21.819 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T14:39:21.819 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T14:39:21.819 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:39:21.883 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-2:19.3.0-4872.g0d61375b.el9.x86_64 37/37 2024-09-11T14:39:23.241 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-osd-2:19.3.0-4872.g0d61375b.el9.x86_64 34/37 2024-09-11T14:39:23.269 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-osd-2:19.3.0-4872.g0d61375b.el9.x86_64 34/37 2024-09-11T14:39:23.269 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:39:23.269 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-11T14:39:23.269 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T14:39:23.269 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T14:39:23.269 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:39:23.808 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mon-2:19.3.0-4872.g0d61375b.el9.x86_64 35/37 2024-09-11T14:39:23.832 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mon-2:19.3.0-4872.g0d61375b.el9.x86_64 35/37 2024-09-11T14:39:23.832 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:39:23.832 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-11T14:39:23.832 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T14:39:23.832 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T14:39:23.832 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:39:24.076 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mds-2:19.3.0-4872.g0d61375b.el9.x86_64 36/37 2024-09-11T14:39:24.099 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mds-2:19.3.0-4872.g0d61375b.el9.x86_64 36/37 2024-09-11T14:39:24.099 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:39:24.099 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-11T14:39:24.100 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T14:39:24.100 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T14:39:24.100 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:39:24.173 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-2:19.3.0-4872.g0d61375b.el9.x86_64 37/37 2024-09-11T14:39:28.950 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-2:19.3.0-4872.g0d61375b.el9.x86_64 37/37 2024-09-11T14:39:28.950 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-2:19.3.0-4872.g0d61375b.el9.x86_64 1/37 2024-09-11T14:39:28.950 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mds-2:19.3.0-4872.g0d61375b.el9.x86_64 2/37 2024-09-11T14:39:28.950 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-2:19.3.0-4872.g0d61375b.el9.x86_64 3/37 2024-09-11T14:39:28.950 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mon-2:19.3.0-4872.g0d61375b.el9.x86_64 4/37 2024-09-11T14:39:28.950 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-osd-2:19.3.0-4872.g0d61375b.el9.x86_64 5/37 2024-09-11T14:39:28.951 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4872.g0d61375b.el9. 6/37 2024-09-11T14:39:28.951 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-11T14:39:28.951 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-11T14:39:28.951 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-11T14:39:28.951 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-11T14:39:28.951 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-11T14:39:28.951 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-11T14:39:28.951 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-11T14:39:28.951 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-11T14:39:28.951 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-11T14:39:28.951 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-11T14:39:28.951 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-11T14:39:28.952 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-11T14:39:28.952 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-11T14:39:28.952 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-11T14:39:28.952 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-11T14:39:28.952 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-11T14:39:28.952 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-11T14:39:28.952 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-11T14:39:28.952 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-11T14:39:28.952 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-11T14:39:28.952 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-11T14:39:28.952 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-11T14:39:28.952 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-11T14:39:28.953 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-11T14:39:28.953 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-11T14:39:28.953 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-11T14:39:28.954 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-11T14:39:28.954 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-11T14:39:28.954 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-11T14:39:28.954 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-11T14:39:30.009 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-11T14:39:30.009 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:39:30.009 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-11T14:39:30.009 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:39:30.009 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:39:30.009 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:39:30.009 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:39:30.009 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:39:30.010 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:39:30.010 INFO:teuthology.orchestra.run.smithi203.stdout: lua-5.4.4-4.el9.x86_64 2024-09-11T14:39:30.010 INFO:teuthology.orchestra.run.smithi203.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-11T14:39:30.010 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-11T14:39:30.010 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-11T14:39:30.010 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-11T14:39:30.010 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-11T14:39:30.010 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-11T14:39:30.010 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-11T14:39:30.010 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-11T14:39:30.010 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-11T14:39:30.011 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-11T14:39:30.011 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-11T14:39:30.011 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-11T14:39:30.011 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-11T14:39:30.011 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-11T14:39:30.011 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-11T14:39:30.011 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-11T14:39:30.011 INFO:teuthology.orchestra.run.smithi203.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-11T14:39:30.011 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-11T14:39:30.011 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply-3.11-14.el9.noarch 2024-09-11T14:39:30.011 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-11T14:39:30.011 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-11T14:39:30.012 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-11T14:39:30.012 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-11T14:39:30.012 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-11T14:39:30.012 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-11T14:39:30.012 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-11T14:39:30.012 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-11T14:39:30.012 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-11T14:39:30.012 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-11T14:39:30.012 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-11T14:39:30.012 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:39:30.012 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-11T14:39:30.334 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-base 2024-09-11T14:39:30.898 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:22 ago on Wed 11 Sep 2024 02:38:08 PM UTC. 2024-09-11T14:39:30.988 INFO:teuthology.orchestra.run.smithi203.stdout:Package ceph-base-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:39:31.020 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-2:19.3.0-4872.g0d61375b.el9.x86_64 37/37 2024-09-11T14:39:31.020 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-2:19.3.0-4872.g0d61375b.el9.x86_64 1/37 2024-09-11T14:39:31.020 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mds-2:19.3.0-4872.g0d61375b.el9.x86_64 2/37 2024-09-11T14:39:31.020 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-2:19.3.0-4872.g0d61375b.el9.x86_64 3/37 2024-09-11T14:39:31.020 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mon-2:19.3.0-4872.g0d61375b.el9.x86_64 4/37 2024-09-11T14:39:31.020 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-osd-2:19.3.0-4872.g0d61375b.el9.x86_64 5/37 2024-09-11T14:39:31.020 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4872.g0d61375b.el9. 6/37 2024-09-11T14:39:31.020 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-11T14:39:31.020 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-11T14:39:31.020 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-11T14:39:31.021 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-11T14:39:31.021 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-11T14:39:31.021 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-11T14:39:31.021 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-11T14:39:31.021 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-11T14:39:31.021 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-11T14:39:31.021 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-11T14:39:31.021 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-11T14:39:31.021 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-11T14:39:31.021 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-11T14:39:31.021 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-11T14:39:31.021 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-11T14:39:31.022 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-11T14:39:31.022 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-11T14:39:31.022 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-11T14:39:31.022 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-11T14:39:31.022 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-11T14:39:31.022 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-11T14:39:31.022 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-11T14:39:31.022 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-11T14:39:31.022 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-11T14:39:31.022 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-11T14:39:31.023 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-11T14:39:31.023 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-11T14:39:31.023 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-11T14:39:31.023 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-11T14:39:31.023 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-11T14:39:31.031 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-11T14:39:31.032 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-09-11T14:39:31.032 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-11T14:39:31.107 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install cephadm 2024-09-11T14:39:31.669 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:23 ago on Wed 11 Sep 2024 02:38:08 PM UTC. 2024-09-11T14:39:31.801 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-11T14:39:31.802 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:39:31.802 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-09-11T14:39:31.802 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:39:31.802 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-11T14:39:31.802 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 784 k 2024-09-11T14:39:31.802 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:39:31.802 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-11T14:39:31.802 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:39:31.802 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-09-11T14:39:31.803 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:39:31.803 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 784 k 2024-09-11T14:39:31.803 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 791 k 2024-09-11T14:39:31.803 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-11T14:39:31.947 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-11T14:39:31.947 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:39:31.948 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T14:39:31.948 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:39:31.948 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:39:31.948 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:39:31.948 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:39:31.948 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:39:31.948 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:39:31.948 INFO:teuthology.orchestra.run.smithi181.stdout: lua-5.4.4-4.el9.x86_64 2024-09-11T14:39:31.948 INFO:teuthology.orchestra.run.smithi181.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-11T14:39:31.948 INFO:teuthology.orchestra.run.smithi181.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-11T14:39:31.948 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-11T14:39:31.948 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-11T14:39:31.949 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-11T14:39:31.949 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-11T14:39:31.949 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-11T14:39:31.949 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-11T14:39:31.949 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-11T14:39:31.949 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-11T14:39:31.949 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-11T14:39:31.949 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-11T14:39:31.949 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-11T14:39:31.949 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-11T14:39:31.949 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-11T14:39:31.949 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-11T14:39:31.949 INFO:teuthology.orchestra.run.smithi181.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-11T14:39:31.950 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-11T14:39:31.950 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ply-3.11-14.el9.noarch 2024-09-11T14:39:31.950 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-11T14:39:31.950 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-11T14:39:31.950 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-11T14:39:31.950 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-11T14:39:31.950 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-11T14:39:31.950 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-11T14:39:31.950 INFO:teuthology.orchestra.run.smithi181.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-11T14:39:31.950 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-11T14:39:31.950 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-11T14:39:31.950 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-11T14:39:31.950 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-11T14:39:31.951 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:39:31.951 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T14:39:32.266 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-base 2024-09-11T14:39:32.348 INFO:teuthology.orchestra.run.smithi203.stdout:cephadm-19.3.0-4872.g0d61375b.el9.noarch.rpm 1.4 MB/s | 784 kB 00:00 2024-09-11T14:39:32.348 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:39:32.348 INFO:teuthology.orchestra.run.smithi203.stdout:Total 1.4 MB/s | 784 kB 00:00 2024-09-11T14:39:32.349 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-11T14:39:32.354 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-11T14:39:32.354 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-11T14:39:32.360 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-11T14:39:32.361 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-11T14:39:32.432 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-11T14:39:32.829 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:21 ago on Wed 11 Sep 2024 02:38:11 PM UTC. 2024-09-11T14:39:32.919 INFO:teuthology.orchestra.run.smithi181.stdout:Package ceph-base-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:39:32.926 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:39:32.961 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T14:39:32.962 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-11T14:39:32.962 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T14:39:33.018 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:39:33.030 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install cephadm 2024-09-11T14:39:33.364 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:39:33.590 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:22 ago on Wed 11 Sep 2024 02:38:11 PM UTC. 2024-09-11T14:39:33.625 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:39:33.625 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:39:33.625 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-11T14:39:33.625 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:39:33.625 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:39:33.625 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-11T14:39:33.722 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T14:39:33.723 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:39:33.723 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-11T14:39:33.723 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:39:33.723 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T14:39:33.723 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 784 k 2024-09-11T14:39:33.723 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:39:33.723 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T14:39:33.724 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:39:33.724 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-11T14:39:33.724 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:39:33.724 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 784 k 2024-09-11T14:39:33.724 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 791 k 2024-09-11T14:39:33.724 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T14:39:33.843 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-immutable-object-cache 2024-09-11T14:39:34.019 INFO:teuthology.orchestra.run.smithi181.stdout:cephadm-19.3.0-4872.g0d61375b.el9.noarch.rpm 2.6 MB/s | 784 kB 00:00 2024-09-11T14:39:34.020 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:39:34.020 INFO:teuthology.orchestra.run.smithi181.stdout:Total 2.6 MB/s | 784 kB 00:00 2024-09-11T14:39:34.020 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T14:39:34.027 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T14:39:34.027 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T14:39:34.031 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T14:39:34.032 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T14:39:34.104 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T14:39:34.420 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:26 ago on Wed 11 Sep 2024 02:38:08 PM UTC. 2024-09-11T14:39:34.552 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-11T14:39:34.553 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:39:34.553 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-09-11T14:39:34.553 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:39:34.553 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-11T14:39:34.553 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 158 k 2024-09-11T14:39:34.553 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:39:34.554 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-11T14:39:34.554 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:39:34.554 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-09-11T14:39:34.554 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:39:34.554 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 158 k 2024-09-11T14:39:34.554 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 644 k 2024-09-11T14:39:34.554 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-11T14:39:34.555 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:39:34.665 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:39:34.756 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-immutable-object-cache-19.3.0-4872.g0d6137 780 kB/s | 158 kB 00:00 2024-09-11T14:39:34.757 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:39:34.758 INFO:teuthology.orchestra.run.smithi203.stdout:Total 773 kB/s | 158 kB 00:00 2024-09-11T14:39:34.758 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-11T14:39:34.767 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-11T14:39:34.767 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-11T14:39:34.824 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-11T14:39:34.824 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-11T14:39:34.969 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-11T14:39:35.066 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4872.g0d61375b. 1/1 2024-09-11T14:39:35.066 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:39:35.091 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4872.g0d61375b. 1/1 2024-09-11T14:39:35.092 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:39:35.092 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-11T14:39:35.092 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:39:35.311 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:39:35.311 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:39:35.311 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T14:39:35.311 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:39:35.311 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:39:35.311 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T14:39:35.538 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-immutable-object-cache 2024-09-11T14:39:36.014 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4872.g0d61375b. 1/1 2024-09-11T14:39:36.015 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:39:36.015 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-11T14:39:36.015 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:39:36.015 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:39:36.015 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-11T14:39:36.108 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:25 ago on Wed 11 Sep 2024 02:38:11 PM UTC. 2024-09-11T14:39:36.242 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T14:39:36.243 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:39:36.243 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-09-11T14:39:36.244 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:39:36.244 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T14:39:36.244 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 158 k 2024-09-11T14:39:36.244 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:39:36.244 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T14:39:36.244 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:39:36.245 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-11T14:39:36.245 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:39:36.245 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 158 k 2024-09-11T14:39:36.245 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 644 k 2024-09-11T14:39:36.245 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T14:39:36.250 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr 2024-09-11T14:39:36.441 INFO:teuthology.orchestra.run.smithi181.stdout:ceph-immutable-object-cache-19.3.0-4872.g0d6137 799 kB/s | 158 kB 00:00 2024-09-11T14:39:36.442 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:39:36.442 INFO:teuthology.orchestra.run.smithi181.stdout:Total 793 kB/s | 158 kB 00:00 2024-09-11T14:39:36.442 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T14:39:36.452 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T14:39:36.452 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T14:39:36.509 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T14:39:36.509 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T14:39:36.650 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T14:39:36.701 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4872.g0d61375b. 1/1 2024-09-11T14:39:36.726 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4872.g0d61375b. 1/1 2024-09-11T14:39:36.726 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:39:36.726 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-11T14:39:36.727 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:39:36.823 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:28 ago on Wed 11 Sep 2024 02:38:08 PM UTC. 2024-09-11T14:39:36.918 INFO:teuthology.orchestra.run.smithi203.stdout:Package ceph-mgr-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:39:36.961 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-11T14:39:36.962 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-09-11T14:39:36.962 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-11T14:39:37.043 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-dashboard 2024-09-11T14:39:37.601 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:29 ago on Wed 11 Sep 2024 02:38:08 PM UTC. 2024-09-11T14:39:37.673 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4872.g0d61375b. 1/1 2024-09-11T14:39:37.674 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:39:37.674 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T14:39:37.674 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:39:37.674 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:39:37.674 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T14:39:37.734 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-11T14:39:37.735 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:39:37.735 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-09-11T14:39:37.735 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:39:37.735 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-11T14:39:37.735 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 14 M 2024-09-11T14:39:37.736 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-09-11T14:39:37.736 INFO:teuthology.orchestra.run.smithi203.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-11T14:39:37.736 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 34 k 2024-09-11T14:39:37.736 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 16 k 2024-09-11T14:39:37.736 INFO:teuthology.orchestra.run.smithi203.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-11T14:39:37.736 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-11T14:39:37.737 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-11T14:39:37.737 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-11T14:39:37.737 INFO:teuthology.orchestra.run.smithi203.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-11T14:39:37.737 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-11T14:39:37.737 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-11T14:39:37.738 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:39:37.738 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-11T14:39:37.738 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:39:37.738 INFO:teuthology.orchestra.run.smithi203.stdout:Install 11 Packages 2024-09-11T14:39:37.738 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:39:37.739 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 18 M 2024-09-11T14:39:37.739 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 155 M 2024-09-11T14:39:37.739 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-11T14:39:37.910 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr 2024-09-11T14:39:38.479 INFO:teuthology.orchestra.run.smithi203.stdout:(1/11): ceph-grafana-dashboards-19.3.0-4872.g0d 234 kB/s | 34 kB 00:00 2024-09-11T14:39:38.485 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:27 ago on Wed 11 Sep 2024 02:38:11 PM UTC. 2024-09-11T14:39:38.504 INFO:teuthology.orchestra.run.smithi203.stdout:(2/11): ceph-prometheus-alerts-19.3.0-4872.g0d6 96 kB/s | 16 kB 00:00 2024-09-11T14:39:38.581 INFO:teuthology.orchestra.run.smithi181.stdout:Package ceph-mgr-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:39:38.623 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T14:39:38.624 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-11T14:39:38.624 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T14:39:38.700 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-dashboard 2024-09-11T14:39:38.704 INFO:teuthology.orchestra.run.smithi203.stdout:(3/11): protobuf-compiler-3.14.0-13.el9.x86_64. 4.2 MB/s | 863 kB 00:00 2024-09-11T14:39:38.763 INFO:teuthology.orchestra.run.smithi203.stdout:(4/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 9.2 MB/s | 548 kB 00:00 2024-09-11T14:39:38.788 INFO:teuthology.orchestra.run.smithi203.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 780 kB/s | 19 kB 00:00 2024-09-11T14:39:38.880 INFO:teuthology.orchestra.run.smithi203.stdout:(6/11): python3-grpcio-1.46.7-10.el9.x86_64.rpm 24 MB/s | 2.0 MB 00:00 2024-09-11T14:39:38.914 INFO:teuthology.orchestra.run.smithi203.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 4.3 MB/s | 144 kB 00:00 2024-09-11T14:39:38.940 INFO:teuthology.orchestra.run.smithi203.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-11T14:39:38.965 INFO:teuthology.orchestra.run.smithi203.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 7.4 MB/s | 188 kB 00:00 2024-09-11T14:39:39.123 INFO:teuthology.orchestra.run.smithi203.stdout:(10/11): ceph-mgr-dashboard-19.3.0-4872.g0d6137 18 MB/s | 14 MB 00:00 2024-09-11T14:39:39.260 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:28 ago on Wed 11 Sep 2024 02:38:11 PM UTC. 2024-09-11T14:39:39.393 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T14:39:39.394 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:39:39.394 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-11T14:39:39.394 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:39:39.394 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T14:39:39.394 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 14 M 2024-09-11T14:39:39.394 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-11T14:39:39.394 INFO:teuthology.orchestra.run.smithi181.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-11T14:39:39.394 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 34 k 2024-09-11T14:39:39.395 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 16 k 2024-09-11T14:39:39.395 INFO:teuthology.orchestra.run.smithi181.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-11T14:39:39.395 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-11T14:39:39.395 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-11T14:39:39.395 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-11T14:39:39.395 INFO:teuthology.orchestra.run.smithi181.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-11T14:39:39.395 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-11T14:39:39.395 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-11T14:39:39.395 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:39:39.395 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T14:39:39.395 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:39:39.395 INFO:teuthology.orchestra.run.smithi181.stdout:Install 11 Packages 2024-09-11T14:39:39.395 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:39:39.396 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 18 M 2024-09-11T14:39:39.396 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 155 M 2024-09-11T14:39:39.396 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T14:39:39.465 INFO:teuthology.orchestra.run.smithi203.stdout:(11/11): python3-protobuf-3.14.0-13.el9.noarch. 273 kB/s | 269 kB 00:00 2024-09-11T14:39:39.470 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:39:39.470 INFO:teuthology.orchestra.run.smithi203.stdout:Total 11 MB/s | 18 MB 00:01 2024-09-11T14:39:39.503 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-11T14:39:39.523 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-11T14:39:39.524 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-11T14:39:39.791 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-11T14:39:39.791 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-11T14:39:40.329 INFO:teuthology.orchestra.run.smithi181.stdout:(1/11): ceph-prometheus-alerts-19.3.0-4872.g0d6 107 kB/s | 16 kB 00:00 2024-09-11T14:39:40.342 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-11T14:39:40.354 INFO:teuthology.orchestra.run.smithi181.stdout:(2/11): ceph-grafana-dashboards-19.3.0-4872.g0d 191 kB/s | 34 kB 00:00 2024-09-11T14:39:40.430 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-09-11T14:39:40.468 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-09-11T14:39:40.579 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-09-11T14:39:40.704 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-09-11T14:39:40.765 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-09-11T14:39:40.813 INFO:teuthology.orchestra.run.smithi181.stdout:(3/11): ceph-mgr-dashboard-19.3.0-4872.g0d61375 22 MB/s | 14 MB 00:00 2024-09-11T14:39:40.849 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-09-11T14:39:40.871 INFO:teuthology.orchestra.run.smithi181.stdout:(4/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 9.5 MB/s | 548 kB 00:00 2024-09-11T14:39:40.896 INFO:teuthology.orchestra.run.smithi181.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 777 kB/s | 19 kB 00:00 2024-09-11T14:39:40.900 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-09-11T14:39:40.938 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-09-11T14:39:40.955 INFO:teuthology.orchestra.run.smithi181.stdout:(6/11): python3-grpcio-1.46.7-10.el9.x86_64.rpm 35 MB/s | 2.0 MB 00:00 2024-09-11T14:39:40.978 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4872.g0d61375b.el9 9/11 2024-09-11T14:39:40.980 INFO:teuthology.orchestra.run.smithi181.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.7 MB/s | 144 kB 00:00 2024-09-11T14:39:40.997 INFO:teuthology.orchestra.run.smithi181.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 1.8 MB/s | 31 kB 00:00 2024-09-11T14:39:41.022 INFO:teuthology.orchestra.run.smithi181.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 7.4 MB/s | 188 kB 00:00 2024-09-11T14:39:41.347 INFO:teuthology.orchestra.run.smithi181.stdout:(10/11): python3-protobuf-3.14.0-13.el9.noarch. 264 kB/s | 269 kB 00:01 2024-09-11T14:39:41.489 INFO:teuthology.orchestra.run.smithi181.stdout:(11/11): protobuf-compiler-3.14.0-13.el9.x86_64 761 kB/s | 863 kB 00:01 2024-09-11T14:39:41.494 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:39:41.494 INFO:teuthology.orchestra.run.smithi181.stdout:Total 8.7 MB/s | 18 MB 00:02 2024-09-11T14:39:41.527 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T14:39:41.548 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T14:39:41.548 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T14:39:41.825 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T14:39:41.825 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T14:39:42.373 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T14:39:42.454 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-09-11T14:39:42.493 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-09-11T14:39:42.601 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-09-11T14:39:42.736 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-09-11T14:39:42.796 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-09-11T14:39:42.879 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-09-11T14:39:42.931 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-09-11T14:39:42.960 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-09-11T14:39:42.992 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4872.g0d61375b.el9 9/11 2024-09-11T14:39:44.203 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4872.g0d61375b.el 10/11 2024-09-11T14:39:44.266 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4872.g0d61375b.el9.noa 11/11 2024-09-11T14:39:46.162 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4872.g0d61375b.el 10/11 2024-09-11T14:39:46.182 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4872.g0d61375b.el9.noa 11/11 2024-09-11T14:39:46.182 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4872.g0d61375b.el 1/11 2024-09-11T14:39:46.182 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4872.g0d61375b.el9.noa 2/11 2024-09-11T14:39:46.183 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4872.g0d61375b.el9 3/11 2024-09-11T14:39:46.183 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-09-11T14:39:46.183 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-09-11T14:39:46.183 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-09-11T14:39:46.183 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-09-11T14:39:46.183 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-09-11T14:39:46.183 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-09-11T14:39:46.183 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-09-11T14:39:46.230 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4872.g0d61375b.el9.noa 11/11 2024-09-11T14:39:46.496 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-09-11T14:39:46.496 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:39:46.496 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-11T14:39:46.496 INFO:teuthology.orchestra.run.smithi203.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-11T14:39:46.496 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:39:46.496 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:39:46.497 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:39:46.497 INFO:teuthology.orchestra.run.smithi203.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-11T14:39:46.497 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-11T14:39:46.497 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-11T14:39:46.497 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-11T14:39:46.497 INFO:teuthology.orchestra.run.smithi203.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-11T14:39:46.497 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-11T14:39:46.497 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-11T14:39:46.497 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:39:46.497 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-11T14:39:46.781 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-11T14:39:47.342 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:39 ago on Wed 11 Sep 2024 02:38:08 PM UTC. 2024-09-11T14:39:47.475 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-11T14:39:47.477 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================= 2024-09-11T14:39:47.477 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-09-11T14:39:47.477 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================= 2024-09-11T14:39:47.477 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-11T14:39:47.477 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 7.4 M 2024-09-11T14:39:47.477 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-09-11T14:39:47.477 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-11T14:39:47.477 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-11T14:39:47.477 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-11T14:39:47.477 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-11T14:39:47.477 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-11T14:39:47.477 INFO:teuthology.orchestra.run.smithi203.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-11T14:39:47.478 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-11T14:39:47.478 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-11T14:39:47.478 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-11T14:39:47.478 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-11T14:39:47.478 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-11T14:39:47.478 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:39:47.478 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-11T14:39:47.478 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================= 2024-09-11T14:39:47.478 INFO:teuthology.orchestra.run.smithi203.stdout:Install 12 Packages 2024-09-11T14:39:47.478 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:39:47.479 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 38 M 2024-09-11T14:39:47.479 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 212 M 2024-09-11T14:39:47.479 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-11T14:39:47.990 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4872.g0d61375b.el9.noa 11/11 2024-09-11T14:39:47.991 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4872.g0d61375b.el 1/11 2024-09-11T14:39:47.991 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4872.g0d61375b.el9.noa 2/11 2024-09-11T14:39:47.991 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4872.g0d61375b.el9 3/11 2024-09-11T14:39:47.991 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-09-11T14:39:47.991 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-09-11T14:39:47.991 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-09-11T14:39:47.991 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-09-11T14:39:47.991 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-09-11T14:39:47.991 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-09-11T14:39:47.991 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-09-11T14:39:47.992 INFO:teuthology.orchestra.run.smithi203.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 912 kB/s | 191 kB 00:00 2024-09-11T14:39:48.058 INFO:teuthology.orchestra.run.smithi203.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.8 MB/s | 801 kB 00:00 2024-09-11T14:39:48.233 INFO:teuthology.orchestra.run.smithi203.stdout:(3/12): ceph-mgr-diskprediction-local-19.3.0-48 16 MB/s | 7.4 MB 00:00 2024-09-11T14:39:48.305 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-09-11T14:39:48.305 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:39:48.305 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T14:39:48.305 INFO:teuthology.orchestra.run.smithi181.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-11T14:39:48.305 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:39:48.305 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:39:48.305 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:39:48.305 INFO:teuthology.orchestra.run.smithi181.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-11T14:39:48.305 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-11T14:39:48.306 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-11T14:39:48.306 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-11T14:39:48.306 INFO:teuthology.orchestra.run.smithi181.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-11T14:39:48.306 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-11T14:39:48.306 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-11T14:39:48.306 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:39:48.306 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T14:39:48.616 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-11T14:39:48.700 INFO:teuthology.orchestra.run.smithi203.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 46 kB/s | 33 kB 00:00 2024-09-11T14:39:48.800 INFO:teuthology.orchestra.run.smithi203.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 32 kB/s | 18 kB 00:00 2024-09-11T14:39:49.183 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:38 ago on Wed 11 Sep 2024 02:38:11 PM UTC. 2024-09-11T14:39:49.242 INFO:teuthology.orchestra.run.smithi203.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.5 MB/s | 3.0 MB 00:01 2024-09-11T14:39:49.318 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T14:39:49.319 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-09-11T14:39:49.320 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-11T14:39:49.320 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-09-11T14:39:49.320 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T14:39:49.320 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 7.4 M 2024-09-11T14:39:49.320 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-11T14:39:49.320 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-11T14:39:49.320 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-11T14:39:49.320 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-11T14:39:49.320 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-11T14:39:49.320 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-11T14:39:49.321 INFO:teuthology.orchestra.run.smithi181.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-11T14:39:49.321 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-11T14:39:49.321 INFO:teuthology.orchestra.run.smithi181.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-11T14:39:49.321 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-11T14:39:49.321 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-11T14:39:49.321 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-11T14:39:49.321 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:39:49.321 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T14:39:49.321 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-09-11T14:39:49.321 INFO:teuthology.orchestra.run.smithi181.stdout:Install 12 Packages 2024-09-11T14:39:49.321 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:39:49.322 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 38 M 2024-09-11T14:39:49.322 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 212 M 2024-09-11T14:39:49.322 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T14:39:49.351 INFO:teuthology.orchestra.run.smithi203.stdout:(7/12): openblas-0.3.26-2.el9.x86_64.rpm 59 kB/s | 38 kB 00:00 2024-09-11T14:39:49.612 INFO:teuthology.orchestra.run.smithi181.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.5 MB/s | 191 kB 00:00 2024-09-11T14:39:49.679 INFO:teuthology.orchestra.run.smithi181.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.0 MB/s | 801 kB 00:00 2024-09-11T14:39:50.084 INFO:teuthology.orchestra.run.smithi203.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 3.8 MB/s | 4.9 MB 00:01 2024-09-11T14:39:50.110 INFO:teuthology.orchestra.run.smithi203.stdout:(9/12): python3-devel-3.9.19-8.el9.x86_64.rpm 282 kB/s | 245 kB 00:00 2024-09-11T14:39:50.212 INFO:teuthology.orchestra.run.smithi181.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 54 kB/s | 33 kB 00:00 2024-09-11T14:39:50.619 INFO:teuthology.orchestra.run.smithi203.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.0 MB/s | 5.1 MB 00:01 2024-09-11T14:39:50.738 INFO:teuthology.orchestra.run.smithi181.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 35 kB/s | 18 kB 00:00 2024-09-11T14:39:50.829 INFO:teuthology.orchestra.run.smithi181.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.6 MB/s | 3.0 MB 00:01 2024-09-11T14:39:50.969 INFO:teuthology.orchestra.run.smithi203.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 284 kB/s | 250 kB 00:00 2024-09-11T14:39:51.030 INFO:teuthology.orchestra.run.smithi181.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-48 4.8 MB/s | 7.4 MB 00:01 2024-09-11T14:39:51.438 INFO:teuthology.orchestra.run.smithi181.stdout:(7/12): openblas-0.3.26-2.el9.x86_64.rpm 55 kB/s | 38 kB 00:00 2024-09-11T14:39:51.727 INFO:teuthology.orchestra.run.smithi203.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 10 MB/s | 16 MB 00:01 2024-09-11T14:39:51.729 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:39:51.729 INFO:teuthology.orchestra.run.smithi203.stdout:Total 9.0 MB/s | 38 MB 00:04 2024-09-11T14:39:51.958 INFO:teuthology.orchestra.run.smithi181.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 266 kB/s | 245 kB 00:00 2024-09-11T14:39:52.090 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-11T14:39:52.116 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-11T14:39:52.117 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-11T14:39:52.122 INFO:teuthology.orchestra.run.smithi181.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 3.8 MB/s | 4.9 MB 00:01 2024-09-11T14:39:52.548 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-11T14:39:52.548 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-11T14:39:52.722 INFO:teuthology.orchestra.run.smithi181.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.0 MB/s | 5.1 MB 00:01 2024-09-11T14:39:52.831 INFO:teuthology.orchestra.run.smithi181.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 287 kB/s | 250 kB 00:00 2024-09-11T14:39:53.314 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-11T14:39:53.389 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-11T14:39:53.437 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-11T14:39:53.543 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-11T14:39:53.577 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-11T14:39:53.763 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-11T14:39:53.792 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-11T14:39:53.806 INFO:teuthology.orchestra.run.smithi181.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 9.7 MB/s | 16 MB 00:01 2024-09-11T14:39:53.808 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:39:53.808 INFO:teuthology.orchestra.run.smithi181.stdout:Total 8.5 MB/s | 38 MB 00:04 2024-09-11T14:39:53.869 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-11T14:39:54.168 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T14:39:54.195 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T14:39:54.195 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T14:39:54.534 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-11T14:39:54.620 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T14:39:54.621 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T14:39:54.649 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-11T14:39:55.407 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T14:39:55.475 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-11T14:39:55.512 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-11T14:39:55.612 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-11T14:39:55.672 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-11T14:39:55.858 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-11T14:39:55.904 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-11T14:39:55.913 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-11T14:39:55.981 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-11T14:39:56.635 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-11T14:39:56.743 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-11T14:39:56.938 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-11T14:39:56.969 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4872.g0d613 12/12 2024-09-11T14:39:58.007 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-11T14:39:59.040 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-11T14:39:59.070 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4872.g0d613 12/12 2024-09-11T14:39:59.623 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4872.g0d613 12/12 2024-09-11T14:39:59.623 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4872.g0d613 1/12 2024-09-11T14:39:59.623 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-11T14:39:59.623 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-11T14:39:59.623 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-11T14:39:59.623 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-11T14:39:59.624 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-11T14:39:59.624 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-11T14:39:59.624 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-11T14:39:59.624 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-11T14:39:59.624 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-11T14:39:59.624 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-11T14:40:00.064 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-11T14:40:00.064 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:00.064 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-11T14:40:00.064 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:40:00.064 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-11T14:40:00.065 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-11T14:40:00.065 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-11T14:40:00.065 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-11T14:40:00.065 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-11T14:40:00.065 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-11T14:40:00.065 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-11T14:40:00.065 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-11T14:40:00.065 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-11T14:40:00.065 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-11T14:40:00.065 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-11T14:40:00.065 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:00.065 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-11T14:40:00.502 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-rook 2024-09-11T14:40:01.084 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:53 ago on Wed 11 Sep 2024 02:38:08 PM UTC. 2024-09-11T14:40:01.221 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-11T14:40:01.222 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:40:01.222 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-09-11T14:40:01.222 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:40:01.222 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-11T14:40:01.222 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 49 k 2024-09-11T14:40:01.222 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-09-11T14:40:01.222 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-11T14:40:01.222 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-11T14:40:01.222 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-11T14:40:01.222 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-11T14:40:01.222 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-11T14:40:01.223 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-11T14:40:01.223 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-11T14:40:01.223 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-11T14:40:01.223 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-11T14:40:01.223 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-11T14:40:01.223 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-11T14:40:01.223 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-11T14:40:01.223 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:01.223 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-11T14:40:01.223 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:40:01.223 INFO:teuthology.orchestra.run.smithi203.stdout:Install 13 Packages 2024-09-11T14:40:01.223 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:01.224 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 2.2 M 2024-09-11T14:40:01.224 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 26 M 2024-09-11T14:40:01.224 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-11T14:40:01.464 INFO:teuthology.orchestra.run.smithi203.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 175 kB/s | 19 kB 00:00 2024-09-11T14:40:01.489 INFO:teuthology.orchestra.run.smithi203.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 190 kB/s | 26 kB 00:00 2024-09-11T14:40:01.548 INFO:teuthology.orchestra.run.smithi203.stdout:(3/13): ceph-mgr-rook-19.3.0-4872.g0d61375b.el9 255 kB/s | 49 kB 00:00 2024-09-11T14:40:01.589 INFO:teuthology.orchestra.run.smithi203.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 159 kB 00:00 2024-09-11T14:40:01.615 INFO:teuthology.orchestra.run.smithi203.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 222 kB 00:00 2024-09-11T14:40:01.657 INFO:teuthology.orchestra.run.smithi203.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 769 kB/s | 32 kB 00:00 2024-09-11T14:40:01.671 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4872.g0d613 12/12 2024-09-11T14:40:01.671 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4872.g0d613 1/12 2024-09-11T14:40:01.671 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-11T14:40:01.671 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-11T14:40:01.671 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-11T14:40:01.671 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-11T14:40:01.671 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-11T14:40:01.671 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-11T14:40:01.672 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-11T14:40:01.672 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-11T14:40:01.672 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-11T14:40:01.672 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-11T14:40:01.682 INFO:teuthology.orchestra.run.smithi203.stdout:(7/13): python3-certifi-2023.05.07-4.el9.noarch 567 kB/s | 14 kB 00:00 2024-09-11T14:40:01.716 INFO:teuthology.orchestra.run.smithi203.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.6 MB/s | 279 kB 00:00 2024-09-11T14:40:01.741 INFO:teuthology.orchestra.run.smithi203.stdout:(9/13): python3-requests-oauthlib-1.3.0-12.el9. 355 kB/s | 54 kB 00:00 2024-09-11T14:40:01.783 INFO:teuthology.orchestra.run.smithi203.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.1 MB/s | 220 kB 00:00 2024-09-11T14:40:01.808 INFO:teuthology.orchestra.run.smithi203.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.8 MB/s | 90 kB 00:00 2024-09-11T14:40:01.833 INFO:teuthology.orchestra.run.smithi203.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 692 kB/s | 59 kB 00:00 2024-09-11T14:40:01.867 INFO:teuthology.orchestra.run.smithi203.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-09-11T14:40:01.869 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:40:01.870 INFO:teuthology.orchestra.run.smithi203.stdout:Total 3.4 MB/s | 2.2 MB 00:00 2024-09-11T14:40:01.935 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-11T14:40:01.949 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-11T14:40:01.950 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-11T14:40:02.034 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-11T14:40:02.034 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:02.034 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T14:40:02.034 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:40:02.034 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-11T14:40:02.034 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-11T14:40:02.034 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-11T14:40:02.034 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-11T14:40:02.034 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-11T14:40:02.034 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-11T14:40:02.034 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-11T14:40:02.034 INFO:teuthology.orchestra.run.smithi181.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-11T14:40:02.035 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-11T14:40:02.035 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-11T14:40:02.035 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-11T14:40:02.035 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:02.035 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T14:40:02.039 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-11T14:40:02.040 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-11T14:40:02.432 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-rook 2024-09-11T14:40:02.553 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-11T14:40:02.725 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-11T14:40:02.785 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-11T14:40:02.857 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-11T14:40:02.895 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-11T14:40:02.937 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-11T14:40:03.005 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:52 ago on Wed 11 Sep 2024 02:38:11 PM UTC. 2024-09-11T14:40:03.040 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-11T14:40:03.141 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T14:40:03.142 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:40:03.142 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-11T14:40:03.142 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:40:03.142 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T14:40:03.142 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 49 k 2024-09-11T14:40:03.142 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-11T14:40:03.142 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-11T14:40:03.143 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-11T14:40:03.143 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-11T14:40:03.143 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-11T14:40:03.143 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-11T14:40:03.143 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-11T14:40:03.143 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-11T14:40:03.143 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-11T14:40:03.143 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-11T14:40:03.143 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-11T14:40:03.143 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-11T14:40:03.143 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-11T14:40:03.144 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:03.144 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T14:40:03.144 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:40:03.144 INFO:teuthology.orchestra.run.smithi181.stdout:Install 13 Packages 2024-09-11T14:40:03.144 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:03.145 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 2.2 M 2024-09-11T14:40:03.145 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 26 M 2024-09-11T14:40:03.145 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T14:40:03.158 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-11T14:40:03.205 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-11T14:40:03.429 INFO:teuthology.orchestra.run.smithi181.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 248 kB/s | 19 kB 00:00 2024-09-11T14:40:03.454 INFO:teuthology.orchestra.run.smithi181.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 250 kB/s | 26 kB 00:00 2024-09-11T14:40:03.513 INFO:teuthology.orchestra.run.smithi181.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 222 kB 00:00 2024-09-11T14:40:03.538 INFO:teuthology.orchestra.run.smithi181.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 159 kB 00:00 2024-09-11T14:40:03.563 INFO:teuthology.orchestra.run.smithi181.stdout:(5/13): ceph-mgr-rook-19.3.0-4872.g0d61375b.el9 233 kB/s | 49 kB 00:00 2024-09-11T14:40:03.589 INFO:teuthology.orchestra.run.smithi181.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-11T14:40:03.614 INFO:teuthology.orchestra.run.smithi181.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 279 kB 00:00 2024-09-11T14:40:03.664 INFO:teuthology.orchestra.run.smithi181.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 321 kB/s | 32 kB 00:00 2024-09-11T14:40:03.690 INFO:teuthology.orchestra.run.smithi181.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 140 kB/s | 14 kB 00:00 2024-09-11T14:40:03.715 INFO:teuthology.orchestra.run.smithi181.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-11T14:40:03.740 INFO:teuthology.orchestra.run.smithi181.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.9 MB/s | 220 kB 00:00 2024-09-11T14:40:03.765 INFO:teuthology.orchestra.run.smithi181.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-11T14:40:03.807 INFO:teuthology.orchestra.run.smithi181.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-09-11T14:40:03.810 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:40:03.810 INFO:teuthology.orchestra.run.smithi181.stdout:Total 3.3 MB/s | 2.2 MB 00:00 2024-09-11T14:40:03.872 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T14:40:03.887 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T14:40:03.887 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T14:40:03.932 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-11T14:40:03.979 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T14:40:03.979 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T14:40:03.987 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-11T14:40:04.029 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-11T14:40:04.094 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-11T14:40:04.124 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-rook-2:19.3.0-4872.g0d61375b.el9.noarch 13/13 2024-09-11T14:40:04.490 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T14:40:04.655 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-11T14:40:04.709 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-11T14:40:04.773 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-11T14:40:04.809 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-11T14:40:04.850 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-11T14:40:04.911 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4872.g0d61375b.el9.noarch 13/13 2024-09-11T14:40:04.911 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4872.g0d61375b.el9.noarch 1/13 2024-09-11T14:40:04.911 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-11T14:40:04.912 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-11T14:40:04.912 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-11T14:40:04.912 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-11T14:40:04.912 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-11T14:40:04.912 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-11T14:40:04.912 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-11T14:40:04.912 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-11T14:40:04.912 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-11T14:40:04.912 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-11T14:40:04.912 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-11T14:40:04.952 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-11T14:40:05.063 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-11T14:40:05.119 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-11T14:40:05.253 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-11T14:40:05.253 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:05.253 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-11T14:40:05.253 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:40:05.253 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-11T14:40:05.253 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-11T14:40:05.253 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-11T14:40:05.254 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-11T14:40:05.254 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-11T14:40:05.254 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-11T14:40:05.254 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-11T14:40:05.254 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-11T14:40:05.254 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-11T14:40:05.254 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-11T14:40:05.254 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-11T14:40:05.254 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-11T14:40:05.254 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:05.254 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-11T14:40:05.536 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-cephadm 2024-09-11T14:40:05.836 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-11T14:40:05.893 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-11T14:40:05.925 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-11T14:40:05.973 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-11T14:40:05.995 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-rook-2:19.3.0-4872.g0d61375b.el9.noarch 13/13 2024-09-11T14:40:06.101 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:58 ago on Wed 11 Sep 2024 02:38:08 PM UTC. 2024-09-11T14:40:06.237 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-11T14:40:06.238 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:40:06.238 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-09-11T14:40:06.238 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:40:06.238 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-11T14:40:06.238 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 154 k 2024-09-11T14:40:06.238 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-09-11T14:40:06.239 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-11T14:40:06.239 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-11T14:40:06.239 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-11T14:40:06.239 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-11T14:40:06.239 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-11T14:40:06.239 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:06.239 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-11T14:40:06.239 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:40:06.239 INFO:teuthology.orchestra.run.smithi203.stdout:Install 6 Packages 2024-09-11T14:40:06.239 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:06.240 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 7.0 M 2024-09-11T14:40:06.240 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 33 M 2024-09-11T14:40:06.240 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-11T14:40:06.555 INFO:teuthology.orchestra.run.smithi203.stdout:(1/6): ceph-mgr-cephadm-19.3.0-4872.g0d61375b.e 727 kB/s | 154 kB 00:00 2024-09-11T14:40:06.614 INFO:teuthology.orchestra.run.smithi203.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.2 MB/s | 548 kB 00:00 2024-09-11T14:40:06.639 INFO:teuthology.orchestra.run.smithi203.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.2 MB/s | 58 kB 00:00 2024-09-11T14:40:06.664 INFO:teuthology.orchestra.run.smithi203.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-11T14:40:06.690 INFO:teuthology.orchestra.run.smithi203.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 716 kB/s | 248 kB 00:00 2024-09-11T14:40:06.776 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4872.g0d61375b.el9.noarch 13/13 2024-09-11T14:40:06.777 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4872.g0d61375b.el9.noarch 1/13 2024-09-11T14:40:06.777 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-11T14:40:06.777 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-11T14:40:06.777 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-11T14:40:06.777 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-11T14:40:06.777 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-11T14:40:06.777 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-11T14:40:06.777 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-11T14:40:06.777 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-11T14:40:06.777 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-11T14:40:06.777 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-11T14:40:06.777 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-11T14:40:07.102 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-11T14:40:07.102 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:07.102 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T14:40:07.102 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:40:07.103 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-11T14:40:07.103 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-11T14:40:07.103 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-11T14:40:07.103 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-11T14:40:07.103 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-11T14:40:07.103 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-11T14:40:07.103 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-11T14:40:07.103 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-11T14:40:07.103 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-11T14:40:07.103 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-11T14:40:07.103 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-11T14:40:07.103 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-11T14:40:07.103 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:07.104 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T14:40:07.371 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-cephadm 2024-09-11T14:40:07.781 INFO:teuthology.orchestra.run.smithi203.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.1 MB/s | 6.0 MB 00:01 2024-09-11T14:40:07.783 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:40:07.783 INFO:teuthology.orchestra.run.smithi203.stdout:Total 4.5 MB/s | 7.0 MB 00:01 2024-09-11T14:40:07.854 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-11T14:40:07.865 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-11T14:40:07.865 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-11T14:40:07.937 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:56 ago on Wed 11 Sep 2024 02:38:11 PM UTC. 2024-09-11T14:40:07.968 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-11T14:40:07.968 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-11T14:40:08.072 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T14:40:08.072 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:40:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-11T14:40:08.073 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:40:08.073 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T14:40:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 154 k 2024-09-11T14:40:08.073 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-11T14:40:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-11T14:40:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-11T14:40:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-11T14:40:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-11T14:40:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-11T14:40:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:08.073 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T14:40:08.074 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:40:08.074 INFO:teuthology.orchestra.run.smithi181.stdout:Install 6 Packages 2024-09-11T14:40:08.074 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:08.074 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 7.0 M 2024-09-11T14:40:08.074 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 33 M 2024-09-11T14:40:08.074 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T14:40:08.282 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-11T14:40:08.457 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-11T14:40:08.531 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-11T14:40:08.609 INFO:teuthology.orchestra.run.smithi181.stdout:(1/6): ceph-mgr-cephadm-19.3.0-4872.g0d61375b.e 697 kB/s | 154 kB 00:00 2024-09-11T14:40:08.659 INFO:teuthology.orchestra.run.smithi181.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-11T14:40:08.685 INFO:teuthology.orchestra.run.smithi181.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-11T14:40:08.710 INFO:teuthology.orchestra.run.smithi181.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-11T14:40:09.035 INFO:teuthology.orchestra.run.smithi181.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 383 kB/s | 248 kB 00:00 2024-09-11T14:40:09.072 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-11T14:40:09.162 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-11T14:40:09.230 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-11T14:40:09.255 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 6/6 2024-09-11T14:40:10.122 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 6/6 2024-09-11T14:40:10.122 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 1/6 2024-09-11T14:40:10.123 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-11T14:40:10.123 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-11T14:40:10.123 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-11T14:40:10.123 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-11T14:40:10.401 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-11T14:40:10.401 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:10.401 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-11T14:40:10.401 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:40:10.401 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-11T14:40:10.401 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-11T14:40:10.401 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-11T14:40:10.401 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-11T14:40:10.401 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-11T14:40:10.401 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:10.402 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-11T14:40:10.708 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-fuse 2024-09-11T14:40:11.077 INFO:teuthology.orchestra.run.smithi181.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 2.2 MB/s | 6.0 MB 00:02 2024-09-11T14:40:11.079 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:40:11.079 INFO:teuthology.orchestra.run.smithi181.stdout:Total 2.3 MB/s | 7.0 MB 00:03 2024-09-11T14:40:11.152 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T14:40:11.164 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T14:40:11.164 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T14:40:11.264 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T14:40:11.264 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T14:40:11.267 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:03 ago on Wed 11 Sep 2024 02:38:08 PM UTC. 2024-09-11T14:40:11.400 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-11T14:40:11.401 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:40:11.401 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-09-11T14:40:11.401 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:40:11.401 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-11T14:40:11.401 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 884 k 2024-09-11T14:40:11.401 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-09-11T14:40:11.402 INFO:teuthology.orchestra.run.smithi203.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-11T14:40:11.402 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:11.402 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-11T14:40:11.402 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:40:11.402 INFO:teuthology.orchestra.run.smithi203.stdout:Install 2 Packages 2024-09-11T14:40:11.402 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:11.402 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 964 k 2024-09-11T14:40:11.402 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 4.1 M 2024-09-11T14:40:11.402 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-11T14:40:11.568 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T14:40:11.710 INFO:teuthology.orchestra.run.smithi203.stdout:(1/2): ceph-fuse-19.3.0-4872.g0d61375b.el9.x86_ 3.3 MB/s | 884 kB 00:00 2024-09-11T14:40:11.725 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-11T14:40:11.779 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-11T14:40:12.177 INFO:teuthology.orchestra.run.smithi203.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 109 kB/s | 80 kB 00:00 2024-09-11T14:40:12.178 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:40:12.178 INFO:teuthology.orchestra.run.smithi203.stdout:Total 1.2 MB/s | 964 kB 00:00 2024-09-11T14:40:12.191 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-11T14:40:12.216 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-11T14:40:12.216 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-11T14:40:12.296 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-11T14:40:12.297 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-11T14:40:12.300 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-11T14:40:12.389 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-11T14:40:12.457 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-11T14:40:12.482 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 6/6 2024-09-11T14:40:12.483 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-11T14:40:12.634 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-11T14:40:12.710 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 2/2 2024-09-11T14:40:13.286 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 6/6 2024-09-11T14:40:13.286 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 1/6 2024-09-11T14:40:13.286 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-11T14:40:13.286 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-11T14:40:13.286 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-11T14:40:13.287 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-11T14:40:13.382 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 2/2 2024-09-11T14:40:13.382 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 1/2 2024-09-11T14:40:13.560 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-11T14:40:13.560 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:13.560 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T14:40:13.560 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:40:13.560 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-11T14:40:13.561 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-11T14:40:13.561 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-11T14:40:13.561 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-11T14:40:13.561 INFO:teuthology.orchestra.run.smithi181.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-11T14:40:13.561 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:13.561 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T14:40:13.667 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-11T14:40:13.667 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:13.667 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-11T14:40:13.667 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-11T14:40:13.667 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:13.667 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-11T14:40:13.861 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-fuse 2024-09-11T14:40:13.963 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-volume 2024-09-11T14:40:14.425 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:03 ago on Wed 11 Sep 2024 02:38:11 PM UTC. 2024-09-11T14:40:14.546 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:06 ago on Wed 11 Sep 2024 02:38:08 PM UTC. 2024-09-11T14:40:14.558 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T14:40:14.559 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:40:14.559 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-11T14:40:14.559 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:40:14.559 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T14:40:14.559 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 884 k 2024-09-11T14:40:14.560 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-11T14:40:14.560 INFO:teuthology.orchestra.run.smithi181.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-11T14:40:14.560 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:14.560 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T14:40:14.560 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:40:14.561 INFO:teuthology.orchestra.run.smithi181.stdout:Install 2 Packages 2024-09-11T14:40:14.561 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:14.561 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 964 k 2024-09-11T14:40:14.561 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 4.1 M 2024-09-11T14:40:14.561 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T14:40:14.684 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-11T14:40:14.685 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:40:14.685 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-09-11T14:40:14.685 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:40:14.685 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-11T14:40:14.685 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 285 k 2024-09-11T14:40:14.685 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:14.686 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-11T14:40:14.686 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:40:14.686 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-09-11T14:40:14.686 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:14.686 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 285 k 2024-09-11T14:40:14.686 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 1.3 M 2024-09-11T14:40:14.686 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-11T14:40:14.729 INFO:teuthology.orchestra.run.smithi181.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 664 kB/s | 80 kB 00:00 2024-09-11T14:40:15.162 INFO:teuthology.orchestra.run.smithi181.stdout:(2/2): ceph-fuse-19.3.0-4872.g0d61375b.el9.x86_ 1.6 MB/s | 884 kB 00:00 2024-09-11T14:40:15.162 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:40:15.163 INFO:teuthology.orchestra.run.smithi181.stdout:Total 1.6 MB/s | 964 kB 00:00 2024-09-11T14:40:15.167 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-volume-19.3.0-4872.g0d61375b.el9.noarch.rp 593 kB/s | 285 kB 00:00 2024-09-11T14:40:15.167 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:40:15.168 INFO:teuthology.orchestra.run.smithi203.stdout:Total 591 kB/s | 285 kB 00:00 2024-09-11T14:40:15.168 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-11T14:40:15.174 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T14:40:15.176 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-11T14:40:15.176 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-11T14:40:15.198 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T14:40:15.198 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T14:40:15.205 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-11T14:40:15.206 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-11T14:40:15.278 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T14:40:15.279 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T14:40:15.425 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-11T14:40:15.469 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T14:40:15.494 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-volume-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:40:15.511 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-volume-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:40:15.511 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:40:15.511 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-11T14:40:15.511 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:15.616 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-11T14:40:15.687 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 2/2 2024-09-11T14:40:16.383 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 2/2 2024-09-11T14:40:16.383 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 1/2 2024-09-11T14:40:16.495 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-volume-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:40:16.496 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:16.496 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-11T14:40:16.496 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:40:16.496 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:16.496 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-11T14:40:16.663 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-11T14:40:16.663 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:16.663 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T14:40:16.663 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-11T14:40:16.663 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:16.663 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T14:40:16.816 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librados-devel 2024-09-11T14:40:16.965 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-volume 2024-09-11T14:40:17.393 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:09 ago on Wed 11 Sep 2024 02:38:08 PM UTC. 2024-09-11T14:40:17.533 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-11T14:40:17.534 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:40:17.534 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-09-11T14:40:17.534 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:40:17.534 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-11T14:40:17.534 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 125 k 2024-09-11T14:40:17.535 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:17.535 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-11T14:40:17.535 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:40:17.535 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-09-11T14:40:17.535 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:17.535 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 125 k 2024-09-11T14:40:17.535 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 445 k 2024-09-11T14:40:17.535 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-11T14:40:17.536 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:06 ago on Wed 11 Sep 2024 02:38:11 PM UTC. 2024-09-11T14:40:17.675 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T14:40:17.676 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:40:17.676 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-11T14:40:17.676 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:40:17.676 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T14:40:17.676 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 285 k 2024-09-11T14:40:17.676 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:17.676 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T14:40:17.676 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:40:17.676 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-11T14:40:17.677 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:17.677 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 285 k 2024-09-11T14:40:17.677 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 1.3 M 2024-09-11T14:40:17.677 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T14:40:17.934 INFO:teuthology.orchestra.run.smithi203.stdout:librados-devel-19.3.0-4872.g0d61375b.el9.x86_64 312 kB/s | 125 kB 00:00 2024-09-11T14:40:17.934 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:40:17.935 INFO:teuthology.orchestra.run.smithi203.stdout:Total 311 kB/s | 125 kB 00:00 2024-09-11T14:40:17.935 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-11T14:40:17.940 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-11T14:40:17.940 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-11T14:40:18.005 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-11T14:40:18.005 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-11T14:40:18.154 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-11T14:40:18.159 INFO:teuthology.orchestra.run.smithi181.stdout:ceph-volume-19.3.0-4872.g0d61375b.el9.noarch.rp 591 kB/s | 285 kB 00:00 2024-09-11T14:40:18.159 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:40:18.160 INFO:teuthology.orchestra.run.smithi181.stdout:Total 589 kB/s | 285 kB 00:00 2024-09-11T14:40:18.160 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T14:40:18.169 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T14:40:18.169 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T14:40:18.201 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T14:40:18.201 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T14:40:18.263 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librados-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:40:18.422 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T14:40:18.494 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-volume-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:40:18.511 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-volume-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:40:18.511 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:40:18.511 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-11T14:40:18.511 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:18.683 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:40:18.956 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:40:18.957 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:18.957 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-11T14:40:18.957 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:40:18.957 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:18.957 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-11T14:40:19.254 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install libcephfs2 2024-09-11T14:40:19.476 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-volume-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:40:19.476 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:19.476 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T14:40:19.476 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:40:19.476 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:19.477 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T14:40:19.801 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librados-devel 2024-09-11T14:40:19.826 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:11 ago on Wed 11 Sep 2024 02:38:08 PM UTC. 2024-09-11T14:40:19.918 INFO:teuthology.orchestra.run.smithi203.stdout:Package libcephfs2-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:40:19.965 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-11T14:40:19.966 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-09-11T14:40:19.967 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-11T14:40:20.040 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install libcephfs-devel 2024-09-11T14:40:20.375 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:09 ago on Wed 11 Sep 2024 02:38:11 PM UTC. 2024-09-11T14:40:20.513 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T14:40:20.513 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:40:20.513 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-09-11T14:40:20.513 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:40:20.513 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T14:40:20.513 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 125 k 2024-09-11T14:40:20.513 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:20.513 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T14:40:20.513 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:40:20.514 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-11T14:40:20.514 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:20.514 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 125 k 2024-09-11T14:40:20.514 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 445 k 2024-09-11T14:40:20.514 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T14:40:20.605 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:12 ago on Wed 11 Sep 2024 02:38:08 PM UTC. 2024-09-11T14:40:20.740 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-11T14:40:20.741 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:40:20.741 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-09-11T14:40:20.741 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:40:20.741 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-11T14:40:20.741 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 32 k 2024-09-11T14:40:20.742 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:20.742 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-11T14:40:20.742 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:40:20.742 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-09-11T14:40:20.742 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:20.742 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 32 k 2024-09-11T14:40:20.742 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 142 k 2024-09-11T14:40:20.742 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-11T14:40:20.898 INFO:teuthology.orchestra.run.smithi181.stdout:librados-devel-19.3.0-4872.g0d61375b.el9.x86_64 325 kB/s | 125 kB 00:00 2024-09-11T14:40:20.898 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:40:20.898 INFO:teuthology.orchestra.run.smithi181.stdout:Total 324 kB/s | 125 kB 00:00 2024-09-11T14:40:20.899 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T14:40:20.904 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T14:40:20.904 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T14:40:20.969 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T14:40:20.969 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T14:40:21.031 INFO:teuthology.orchestra.run.smithi203.stdout:libcephfs-devel-19.3.0-4872.g0d61375b.el9.x86_6 110 kB/s | 32 kB 00:00 2024-09-11T14:40:21.032 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:40:21.032 INFO:teuthology.orchestra.run.smithi203.stdout:Total 109 kB/s | 32 kB 00:00 2024-09-11T14:40:21.032 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-11T14:40:21.035 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-11T14:40:21.035 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-11T14:40:21.055 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-11T14:40:21.055 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-11T14:40:21.126 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T14:40:21.144 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-11T14:40:21.230 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librados-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:40:21.253 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:40:21.620 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:40:21.700 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:40:21.845 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:40:21.846 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:21.846 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-11T14:40:21.846 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:40:21.846 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:21.846 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-11T14:40:21.945 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:40:21.945 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:21.945 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T14:40:21.945 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:40:21.945 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:21.945 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T14:40:22.066 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librados2 2024-09-11T14:40:22.221 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install libcephfs2 2024-09-11T14:40:22.632 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:14 ago on Wed 11 Sep 2024 02:38:08 PM UTC. 2024-09-11T14:40:22.724 INFO:teuthology.orchestra.run.smithi203.stdout:Package librados2-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:40:22.768 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-11T14:40:22.768 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-09-11T14:40:22.769 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-11T14:40:22.797 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:11 ago on Wed 11 Sep 2024 02:38:11 PM UTC. 2024-09-11T14:40:22.849 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librbd1 2024-09-11T14:40:22.889 INFO:teuthology.orchestra.run.smithi181.stdout:Package libcephfs2-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:40:22.934 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T14:40:22.936 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-11T14:40:22.936 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T14:40:23.050 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install libcephfs-devel 2024-09-11T14:40:23.412 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:15 ago on Wed 11 Sep 2024 02:38:08 PM UTC. 2024-09-11T14:40:23.502 INFO:teuthology.orchestra.run.smithi203.stdout:Package librbd1-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:40:23.545 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-11T14:40:23.545 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-09-11T14:40:23.546 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-11T14:40:23.617 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:12 ago on Wed 11 Sep 2024 02:38:11 PM UTC. 2024-09-11T14:40:23.634 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rados 2024-09-11T14:40:23.753 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T14:40:23.753 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:40:23.753 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-09-11T14:40:23.754 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:40:23.754 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T14:40:23.754 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 32 k 2024-09-11T14:40:23.754 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:23.754 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T14:40:23.754 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:40:23.754 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-11T14:40:23.754 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:23.754 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 32 k 2024-09-11T14:40:23.754 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 142 k 2024-09-11T14:40:23.754 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T14:40:23.943 INFO:teuthology.orchestra.run.smithi181.stdout:libcephfs-devel-19.3.0-4872.g0d61375b.el9.x86_6 168 kB/s | 32 kB 00:00 2024-09-11T14:40:23.943 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:40:23.943 INFO:teuthology.orchestra.run.smithi181.stdout:Total 167 kB/s | 32 kB 00:00 2024-09-11T14:40:23.944 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T14:40:23.947 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T14:40:23.947 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T14:40:23.969 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T14:40:23.969 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T14:40:24.057 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T14:40:24.164 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:40:24.204 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:16 ago on Wed 11 Sep 2024 02:38:08 PM UTC. 2024-09-11T14:40:24.294 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-rados-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:40:24.338 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-11T14:40:24.338 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-09-11T14:40:24.338 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-11T14:40:24.418 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rgw 2024-09-11T14:40:24.528 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:40:24.757 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:40:24.757 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:24.757 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T14:40:24.757 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:40:24.757 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:24.757 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T14:40:24.956 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librados2 2024-09-11T14:40:24.976 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:16 ago on Wed 11 Sep 2024 02:38:08 PM UTC. 2024-09-11T14:40:25.066 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-rgw-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:40:25.110 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-11T14:40:25.111 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-09-11T14:40:25.111 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-11T14:40:25.197 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-cephfs 2024-09-11T14:40:25.522 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:14 ago on Wed 11 Sep 2024 02:38:11 PM UTC. 2024-09-11T14:40:25.613 INFO:teuthology.orchestra.run.smithi181.stdout:Package librados2-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:40:25.656 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T14:40:25.657 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-11T14:40:25.657 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T14:40:25.729 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librbd1 2024-09-11T14:40:25.761 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:17 ago on Wed 11 Sep 2024 02:38:08 PM UTC. 2024-09-11T14:40:25.851 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-cephfs-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:40:25.895 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-11T14:40:25.896 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-09-11T14:40:25.896 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-11T14:40:25.975 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rbd 2024-09-11T14:40:26.289 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:15 ago on Wed 11 Sep 2024 02:38:11 PM UTC. 2024-09-11T14:40:26.381 INFO:teuthology.orchestra.run.smithi181.stdout:Package librbd1-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:40:26.424 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T14:40:26.425 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-11T14:40:26.425 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T14:40:26.499 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rados 2024-09-11T14:40:26.669 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:18 ago on Wed 11 Sep 2024 02:38:08 PM UTC. 2024-09-11T14:40:26.669 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-rbd-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:40:26.676 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-11T14:40:26.677 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-09-11T14:40:26.677 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-11T14:40:26.759 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-fuse 2024-09-11T14:40:27.230 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:16 ago on Wed 11 Sep 2024 02:38:11 PM UTC. 2024-09-11T14:40:27.319 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rados-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:40:27.336 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:19 ago on Wed 11 Sep 2024 02:38:08 PM UTC. 2024-09-11T14:40:27.363 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T14:40:27.364 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-11T14:40:27.364 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T14:40:27.449 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rgw 2024-09-11T14:40:27.481 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-11T14:40:27.481 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:40:27.481 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-09-11T14:40:27.482 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:40:27.482 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-11T14:40:27.482 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 91 k 2024-09-11T14:40:27.482 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:27.482 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-11T14:40:27.482 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:40:27.482 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-09-11T14:40:27.482 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:27.483 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 91 k 2024-09-11T14:40:27.483 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 321 k 2024-09-11T14:40:27.483 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-11T14:40:27.676 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-fuse-19.3.0-4872.g0d61375b.el9.x86_64.rpm 471 kB/s | 91 kB 00:00 2024-09-11T14:40:27.677 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:40:27.678 INFO:teuthology.orchestra.run.smithi203.stdout:Total 465 kB/s | 91 kB 00:00 2024-09-11T14:40:27.678 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-11T14:40:27.694 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-11T14:40:27.694 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-11T14:40:27.776 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-11T14:40:27.777 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-11T14:40:27.930 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-11T14:40:28.009 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:17 ago on Wed 11 Sep 2024 02:38:11 PM UTC. 2024-09-11T14:40:28.017 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:40:28.099 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rgw-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:40:28.141 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T14:40:28.142 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-11T14:40:28.143 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T14:40:28.221 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-cephfs 2024-09-11T14:40:28.414 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:40:28.666 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:40:28.666 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:28.666 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-11T14:40:28.666 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:40:28.666 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:28.666 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-11T14:40:28.792 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:17 ago on Wed 11 Sep 2024 02:38:11 PM UTC. 2024-09-11T14:40:28.884 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-cephfs-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:40:28.928 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T14:40:28.929 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-11T14:40:28.929 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T14:40:28.936 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-mirror 2024-09-11T14:40:29.016 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rbd 2024-09-11T14:40:29.520 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:21 ago on Wed 11 Sep 2024 02:38:08 PM UTC. 2024-09-11T14:40:29.589 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:18 ago on Wed 11 Sep 2024 02:38:11 PM UTC. 2024-09-11T14:40:29.658 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-11T14:40:29.659 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:40:29.659 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-09-11T14:40:29.659 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:40:29.659 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-11T14:40:29.659 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 3.5 M 2024-09-11T14:40:29.659 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:29.659 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-11T14:40:29.660 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:40:29.660 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-09-11T14:40:29.660 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:29.660 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 3.5 M 2024-09-11T14:40:29.660 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 30 M 2024-09-11T14:40:29.660 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-11T14:40:29.684 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rbd-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:40:29.729 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T14:40:29.730 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-11T14:40:29.730 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T14:40:29.810 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-fuse 2024-09-11T14:40:30.021 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-mirror-19.3.0-4872.g0d61375b.el9.x86_64.rpm 9.7 MB/s | 3.5 MB 00:00 2024-09-11T14:40:30.022 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:40:30.022 INFO:teuthology.orchestra.run.smithi203.stdout:Total 9.7 MB/s | 3.5 MB 00:00 2024-09-11T14:40:30.022 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-11T14:40:30.034 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-11T14:40:30.034 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-11T14:40:30.117 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-11T14:40:30.117 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-11T14:40:30.388 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:19 ago on Wed 11 Sep 2024 02:38:11 PM UTC. 2024-09-11T14:40:30.524 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T14:40:30.524 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:40:30.524 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-09-11T14:40:30.525 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:40:30.525 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T14:40:30.525 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 91 k 2024-09-11T14:40:30.525 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:30.525 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T14:40:30.525 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:40:30.525 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-11T14:40:30.525 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:30.525 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 91 k 2024-09-11T14:40:30.525 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 321 k 2024-09-11T14:40:30.525 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T14:40:30.704 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-11T14:40:30.713 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-fuse-19.3.0-4872.g0d61375b.el9.x86_64.rpm 485 kB/s | 91 kB 00:00 2024-09-11T14:40:30.713 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:40:30.714 INFO:teuthology.orchestra.run.smithi181.stdout:Total 481 kB/s | 91 kB 00:00 2024-09-11T14:40:30.714 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T14:40:30.723 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T14:40:30.724 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T14:40:30.755 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-mirror-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:40:30.781 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:40:30.781 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:40:30.781 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-11T14:40:30.781 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T14:40:30.781 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T14:40:30.781 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:30.787 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T14:40:30.787 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T14:40:30.928 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T14:40:31.028 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:40:31.471 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:40:31.727 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:40:31.727 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:31.728 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T14:40:31.728 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:40:31.728 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:31.728 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T14:40:31.989 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-mirror 2024-09-11T14:40:31.990 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-mirror-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:40:31.990 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:31.990 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-11T14:40:31.990 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:40:31.990 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:31.990 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-11T14:40:32.249 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-nbd 2024-09-11T14:40:32.560 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:21 ago on Wed 11 Sep 2024 02:38:11 PM UTC. 2024-09-11T14:40:32.695 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T14:40:32.696 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:40:32.696 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-09-11T14:40:32.696 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:40:32.696 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T14:40:32.696 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 3.5 M 2024-09-11T14:40:32.696 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:32.696 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T14:40:32.696 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:40:32.696 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-11T14:40:32.696 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:32.697 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 3.5 M 2024-09-11T14:40:32.697 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 30 M 2024-09-11T14:40:32.697 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T14:40:32.832 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:24 ago on Wed 11 Sep 2024 02:38:08 PM UTC. 2024-09-11T14:40:32.969 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-11T14:40:32.970 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:40:32.970 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-09-11T14:40:32.970 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:40:32.970 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-11T14:40:32.970 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 177 k 2024-09-11T14:40:32.970 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:32.970 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-11T14:40:32.970 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-11T14:40:32.970 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-09-11T14:40:32.970 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:32.971 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 177 k 2024-09-11T14:40:32.971 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 685 k 2024-09-11T14:40:32.971 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-11T14:40:33.049 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-mirror-19.3.0-4872.g0d61375b.el9.x86_64.rpm 10 MB/s | 3.5 MB 00:00 2024-09-11T14:40:33.050 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:40:33.051 INFO:teuthology.orchestra.run.smithi181.stdout:Total 9.9 MB/s | 3.5 MB 00:00 2024-09-11T14:40:33.051 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T14:40:33.062 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T14:40:33.062 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T14:40:33.146 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T14:40:33.146 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T14:40:33.193 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-nbd-19.3.0-4872.g0d61375b.el9.x86_64.rpm 795 kB/s | 177 kB 00:00 2024-09-11T14:40:33.194 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:40:33.194 INFO:teuthology.orchestra.run.smithi203.stdout:Total 789 kB/s | 177 kB 00:00 2024-09-11T14:40:33.194 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-11T14:40:33.204 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-11T14:40:33.204 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-11T14:40:33.269 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-11T14:40:33.269 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-11T14:40:33.426 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-11T14:40:33.532 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-nbd-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:40:33.729 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T14:40:33.775 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-mirror-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:40:33.800 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:40:33.800 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:40:33.800 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-11T14:40:33.800 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T14:40:33.800 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T14:40:33.800 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:33.975 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:40:34.231 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-nbd-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:40:34.231 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:34.231 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-11T14:40:34.231 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:40:34.231 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:34.231 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-11T14:40:34.503 DEBUG:teuthology.parallel:result is None 2024-09-11T14:40:34.984 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-mirror-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:40:34.984 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:34.984 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T14:40:34.984 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:40:34.985 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:34.985 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T14:40:35.239 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-nbd 2024-09-11T14:40:35.821 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:24 ago on Wed 11 Sep 2024 02:38:11 PM UTC. 2024-09-11T14:40:35.956 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-11T14:40:35.957 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:40:35.957 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-09-11T14:40:35.957 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:40:35.957 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-11T14:40:35.957 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 177 k 2024-09-11T14:40:35.957 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:35.957 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-11T14:40:35.958 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-11T14:40:35.958 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-11T14:40:35.958 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:35.958 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 177 k 2024-09-11T14:40:35.958 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 685 k 2024-09-11T14:40:35.958 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-11T14:40:36.170 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-nbd-19.3.0-4872.g0d61375b.el9.x86_64.rpm 834 kB/s | 177 kB 00:00 2024-09-11T14:40:36.170 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:40:36.171 INFO:teuthology.orchestra.run.smithi181.stdout:Total 829 kB/s | 177 kB 00:00 2024-09-11T14:40:36.171 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-11T14:40:36.180 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-11T14:40:36.180 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-11T14:40:36.245 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-11T14:40:36.246 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-11T14:40:36.415 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-11T14:40:36.527 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-nbd-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:40:36.979 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:40:37.252 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-nbd-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:40:37.252 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:37.252 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-11T14:40:37.252 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:40:37.252 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:37.252 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-11T14:40:37.523 DEBUG:teuthology.parallel:result is None 2024-09-11T14:40:37.524 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=0d61375b8ab53be4c233f5b93e463edfdff9316a 2024-09-11T14:40:37.662 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-11T14:40:37.685 INFO:teuthology.orchestra.run.smithi181.stdout:19.3.0-4872.g0d61375b.el9 2024-09-11T14:40:37.685 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4872.g0d61375b.el9 2024-09-11T14:40:37.685 INFO:teuthology.task.install:The correct ceph version 19.3.0-4872.g0d61375b is installed. 2024-09-11T14:40:37.687 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=0d61375b8ab53be4c233f5b93e463edfdff9316a 2024-09-11T14:40:37.825 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-11T14:40:37.851 INFO:teuthology.orchestra.run.smithi203.stdout:19.3.0-4872.g0d61375b.el9 2024-09-11T14:40:37.851 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4872.g0d61375b.el9 2024-09-11T14:40:37.851 INFO:teuthology.task.install:The correct ceph version 19.3.0-4872.g0d61375b is installed. 2024-09-11T14:40:37.853 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-11T14:40:37.853 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-11T14:40:37.853 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T14:40:37.879 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-11T14:40:37.880 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T14:40:37.922 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-11T14:40:37.922 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-11T14:40:37.922 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T14:40:37.948 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T14:40:38.013 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-11T14:40:38.013 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T14:40:38.040 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T14:40:38.106 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-11T14:40:38.107 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-11T14:40:38.107 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T14:40:38.132 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T14:40:38.197 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-11T14:40:38.197 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T14:40:38.223 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T14:40:38.290 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-11T14:40:38.290 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-11T14:40:38.290 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T14:40:38.316 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T14:40:38.381 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-11T14:40:38.381 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T14:40:38.407 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T14:40:38.473 INFO:teuthology.run_tasks:Running task ceph... 2024-09-11T14:40:38.554 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-11T14:40:38.555 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 777 /var/log/ceph 2024-09-11T14:40:38.557 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 777 /var/log/ceph 2024-09-11T14:40:38.581 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-11T14:40:38.582 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-11T14:40:38.623 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-11T14:40:38.647 INFO:tasks.ceph:Creating extra log directories... 2024-09-11T14:40:38.647 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-11T14:40:38.688 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-11T14:40:38.722 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-11T14:40:38.722 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'client.0': {'admin socket': '/var/run/ceph/ceph-$name.asok'}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-11T14:40:38.723 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901371', 'branch': 'main', 'description': 'crimson-rados/thrash_simple/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore thrashers/simple thrashosds-health workloads/admin_socket_objecter_requests}', 'email': None, 'first_in_suite': False, 'job_id': '7901371', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client.0': {'admin socket': '/var/run/ceph/ceph-$name.asok'}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a'}}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-matanb-crimson-seastore-logs', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'dbebf1a3db6db7b5579464ee715edcf1b3e9cce2'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'], ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 5813, 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-matanb-crimson-seastore-logs', 'suite_path': '/home/teuthworker/src/github.com_matan-b_ceph_dbebf1a3db6db7b5579464ee715edcf1b3e9cce2/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/matan-b/ceph.git', 'suite_sha1': 'dbebf1a3db6db7b5579464ee715edcf1b3e9cce2', 'targets': {'smithi181.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM8vMyk6v77e2SDon6qWKITXhlQsrMYSD1l9ELy92Nnm9akngN9aF1FVhFjl8cbtFSeyNFSUJxSjp8vvoOlU/us=', 'smithi203.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM07U4gC4pLpBCpPvX+PPOBwSdNsLUuGHL5qCO+OrSXNZhNOI8SQqR3nPk4x8ZIurmHPghFk9t8LeDtNlzKZj8E='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'client.0': {'admin socket': '/var/run/ceph/ceph-$name.asok'}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a', 'cluster': 'ceph'}}, {'thrashosds': {'ceph_objectstore_tool': False, 'chance_down': 0, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400}}, {'radosbench': {'clients': ['client.0'], 'time': 150}}, {'admin_socket': {'client.0': {'objecter_requests': {'test': 'http://git.ceph.com/?p={repo};a=blob_plain;f=src/test/admin_socket/objecter_requests;hb={branch}'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-11_13:56:22', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-11T14:40:38.723 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-11T14:40:38.754 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-11T14:40:38.770 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T14:40:38.810 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T14:40:38.835 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-11T14:40:38.835 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T14:40:38.890 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T14:40:38.890 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_1 2024-09-11T14:40:38.946 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T14:40:38.946 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T14:40:38.947 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-11T14:40:38.947 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T14:40:38.947 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T14:40:38.947 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-11 14:40:34.055010648 +0000 2024-09-11T14:40:38.947 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-11 14:37:29.304190728 +0000 2024-09-11T14:40:38.947 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-11 14:37:29.304190728 +0000 2024-09-11T14:40:38.947 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-11 14:37:29.304190728 +0000 2024-09-11T14:40:38.947 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T14:40:39.009 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-11T14:40:39.010 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-11T14:40:39.010 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.00025941 s, 2.0 MB/s 2024-09-11T14:40:39.011 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T14:40:39.067 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_2 2024-09-11T14:40:39.122 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T14:40:39.122 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T14:40:39.122 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-11T14:40:39.122 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T14:40:39.123 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T14:40:39.123 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-11 14:40:34.055010648 +0000 2024-09-11T14:40:39.123 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-11 14:37:29.632184307 +0000 2024-09-11T14:40:39.123 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-11 14:37:29.632184307 +0000 2024-09-11T14:40:39.123 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-11 14:37:29.632184307 +0000 2024-09-11T14:40:39.123 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T14:40:39.185 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-11T14:40:39.186 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-11T14:40:39.186 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000205387 s, 2.5 MB/s 2024-09-11T14:40:39.187 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T14:40:39.243 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_3 2024-09-11T14:40:39.298 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T14:40:39.298 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T14:40:39.298 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-11T14:40:39.298 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T14:40:39.298 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T14:40:39.299 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-11 14:40:34.056010628 +0000 2024-09-11T14:40:39.299 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-11 14:37:29.945178179 +0000 2024-09-11T14:40:39.299 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-11 14:37:29.945178179 +0000 2024-09-11T14:40:39.299 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-11 14:37:29.945178179 +0000 2024-09-11T14:40:39.299 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T14:40:39.361 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-11T14:40:39.362 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-11T14:40:39.362 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000200596 s, 2.6 MB/s 2024-09-11T14:40:39.363 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T14:40:39.419 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_4 2024-09-11T14:40:39.474 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T14:40:39.474 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T14:40:39.474 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-11T14:40:39.474 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T14:40:39.475 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T14:40:39.475 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-11 14:40:34.056010628 +0000 2024-09-11T14:40:39.475 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-11 14:37:30.264171934 +0000 2024-09-11T14:40:39.475 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-11 14:37:30.264171934 +0000 2024-09-11T14:40:39.475 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-11 14:37:30.264171934 +0000 2024-09-11T14:40:39.475 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T14:40:39.537 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-11T14:40:39.537 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-11T14:40:39.537 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000182471 s, 2.8 MB/s 2024-09-11T14:40:39.538 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T14:40:39.594 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-09-11T14:40:39.595 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-11T14:40:39.595 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T14:40:39.613 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T14:40:39.613 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_1 2024-09-11T14:40:39.670 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T14:40:39.670 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T14:40:39.670 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-11T14:40:39.670 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T14:40:39.670 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T14:40:39.670 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-09-11 14:40:31.021830433 +0000 2024-09-11T14:40:39.670 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-09-11 14:37:29.854695533 +0000 2024-09-11T14:40:39.670 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-09-11 14:37:29.854695533 +0000 2024-09-11T14:40:39.670 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-09-11 14:37:29.854695533 +0000 2024-09-11T14:40:39.670 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T14:40:39.735 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-09-11T14:40:39.735 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-09-11T14:40:39.736 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000277996 s, 1.8 MB/s 2024-09-11T14:40:39.737 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T14:40:39.794 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_2 2024-09-11T14:40:39.851 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T14:40:39.851 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T14:40:39.852 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-11T14:40:39.852 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T14:40:39.852 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T14:40:39.852 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-09-11 14:40:31.022830408 +0000 2024-09-11T14:40:39.852 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-09-11 14:37:30.181687347 +0000 2024-09-11T14:40:39.852 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-09-11 14:37:30.181687347 +0000 2024-09-11T14:40:39.852 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-09-11 14:37:30.181687347 +0000 2024-09-11T14:40:39.852 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T14:40:39.917 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-09-11T14:40:39.917 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-09-11T14:40:39.917 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000211789 s, 2.4 MB/s 2024-09-11T14:40:39.919 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T14:40:39.975 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_3 2024-09-11T14:40:40.032 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T14:40:40.032 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T14:40:40.032 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-11T14:40:40.032 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T14:40:40.032 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T14:40:40.032 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-09-11 14:40:31.023830384 +0000 2024-09-11T14:40:40.032 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-09-11 14:37:30.514679010 +0000 2024-09-11T14:40:40.032 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-09-11 14:37:30.514679010 +0000 2024-09-11T14:40:40.033 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-09-11 14:37:30.514679010 +0000 2024-09-11T14:40:40.033 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T14:40:40.096 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-09-11T14:40:40.096 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-09-11T14:40:40.096 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000207514 s, 2.5 MB/s 2024-09-11T14:40:40.097 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T14:40:40.154 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_4 2024-09-11T14:40:40.210 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T14:40:40.210 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T14:40:40.210 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-11T14:40:40.210 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T14:40:40.211 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T14:40:40.211 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-09-11 14:40:31.023830384 +0000 2024-09-11T14:40:40.211 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-09-11 14:37:30.823671273 +0000 2024-09-11T14:40:40.211 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-09-11 14:37:30.823671273 +0000 2024-09-11T14:40:40.211 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-09-11 14:37:30.823671273 +0000 2024-09-11T14:40:40.211 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T14:40:40.274 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-09-11T14:40:40.274 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-09-11T14:40:40.274 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000185909 s, 2.8 MB/s 2024-09-11T14:40:40.275 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T14:40:40.332 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-09-11T14:40:40.332 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi181.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi203.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-09-11T14:40:40.332 INFO:tasks.ceph:Generating config... 2024-09-11T14:40:40.333 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-11T14:40:40.333 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-09-11T14:40:40.333 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-09-11T14:40:40.333 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-11T14:40:40.333 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-11T14:40:40.334 INFO:tasks.ceph:[osd] debug seastore = 20 2024-09-11T14:40:40.334 INFO:tasks.ceph:[osd] debug seastore backref = 20 2024-09-11T14:40:40.334 INFO:tasks.ceph:[osd] debug seastore cache = 20 2024-09-11T14:40:40.334 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2024-09-11T14:40:40.334 INFO:tasks.ceph:[osd] debug seastore device = 20 2024-09-11T14:40:40.334 INFO:tasks.ceph:[osd] debug seastore epm = 20 2024-09-11T14:40:40.334 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2024-09-11T14:40:40.334 INFO:tasks.ceph:[osd] debug seastore journal = 20 2024-09-11T14:40:40.334 INFO:tasks.ceph:[osd] debug seastore lba = 20 2024-09-11T14:40:40.334 INFO:tasks.ceph:[osd] debug seastore odata = 20 2024-09-11T14:40:40.334 INFO:tasks.ceph:[osd] debug seastore ompap = 20 2024-09-11T14:40:40.334 INFO:tasks.ceph:[osd] debug seastore onode = 20 2024-09-11T14:40:40.334 INFO:tasks.ceph:[osd] debug seastore t = 20 2024-09-11T14:40:40.334 INFO:tasks.ceph:[osd] debug seastore tm = 20 2024-09-11T14:40:40.334 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-09-11T14:40:40.335 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2024-09-11T14:40:40.335 INFO:tasks.ceph:[osd] osd delete sleep = 1 2024-09-11T14:40:40.335 INFO:tasks.ceph:[osd] osd max backfills = 3 2024-09-11T14:40:40.335 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-11T14:40:40.335 INFO:tasks.ceph:[osd] osd objectstore = seastore 2024-09-11T14:40:40.335 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-09-11T14:40:40.335 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-09-11T14:40:40.335 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-11T14:40:40.335 INFO:tasks.ceph:[osd] osd snap trim sleep = 2 2024-09-11T14:40:40.335 INFO:tasks.ceph:[client.0] admin socket = /var/run/ceph/ceph-$name.asok 2024-09-11T14:40:40.335 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-09-11T14:40:40.335 INFO:tasks.ceph:[global] ms client mode = crc 2024-09-11T14:40:40.335 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-09-11T14:40:40.335 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-09-11T14:40:40.335 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-09-11T14:40:40.335 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-09-11T14:40:40.336 INFO:tasks.ceph:[global] ms service mode = crc 2024-09-11T14:40:40.336 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2024-09-11T14:40:40.336 INFO:tasks.ceph:[global] osd_pool_default_size = 3 2024-09-11T14:40:40.336 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-11T14:40:40.336 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-11T14:40:40.336 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-11T14:40:40.336 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-11T14:40:40.336 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-11T14:40:40.336 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-09-11T14:40:40.336 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-11T14:40:40.336 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-11T14:40:40.336 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-11T14:40:40.336 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-09-11T14:40:40.336 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-09-11T14:40:40.336 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-09-11T14:40:40.337 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-09-11T14:40:40.337 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-09-11T14:40:40.337 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-09-11T14:40:40.337 INFO:tasks.ceph:Setting up mon.a... 2024-09-11T14:40:40.337 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-11T14:40:40.397 INFO:teuthology.orchestra.run.smithi181.stdout:creating /etc/ceph/ceph.keyring 2024-09-11T14:40:40.400 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-11T14:40:40.452 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T14:40:40.477 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.181')] 2024-09-11T14:40:40.478 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.181', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'client.0': {'admin socket': '/var/run/ceph/ceph-$name.asok'}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-09-11T14:40:40.480 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-11T14:40:40.480 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T14:40:40.532 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.181 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:40:40.626 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:40:40.624+0000 7f628323ef00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:40:40.626 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:40:40.624+0000 7f628323ef00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:40:40.626 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:40:40.625+0000 7f628323ef00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:40:40.627 INFO:teuthology.orchestra.run.smithi181.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:40:40.627 INFO:teuthology.orchestra.run.smithi181.stdout:monmaptool: generated fsid 0f7e90de-623e-43a2-a836-b493e06f03fc 2024-09-11T14:40:40.627 INFO:teuthology.orchestra.run.smithi181.stdout:setting min_mon_release = quincy 2024-09-11T14:40:40.627 INFO:teuthology.orchestra.run.smithi181.stdout:epoch 0 2024-09-11T14:40:40.627 INFO:teuthology.orchestra.run.smithi181.stdout:fsid 0f7e90de-623e-43a2-a836-b493e06f03fc 2024-09-11T14:40:40.627 INFO:teuthology.orchestra.run.smithi181.stdout:last_changed 2024-09-11T14:40:40.626393+0000 2024-09-11T14:40:40.627 INFO:teuthology.orchestra.run.smithi181.stdout:created 2024-09-11T14:40:40.626393+0000 2024-09-11T14:40:40.628 INFO:teuthology.orchestra.run.smithi181.stdout:min_mon_release 17 (quincy) 2024-09-11T14:40:40.628 INFO:teuthology.orchestra.run.smithi181.stdout:election_strategy: 1 2024-09-11T14:40:40.628 INFO:teuthology.orchestra.run.smithi181.stdout:0: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.a 2024-09-11T14:40:40.628 INFO:teuthology.orchestra.run.smithi181.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-09-11T14:40:40.629 DEBUG:teuthology.orchestra.run.smithi181:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T14:40:40.644 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 0f7e90de-623e-43a2-a836-b493e06f03fc... 2024-09-11T14:40:40.645 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-11T14:40:40.687 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-11T14:40:40.727 INFO:teuthology.orchestra.run.smithi181.stdout:[global] 2024-09-11T14:40:40.728 INFO:teuthology.orchestra.run.smithi181.stdout: chdir = "" 2024-09-11T14:40:40.728 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T14:40:40.728 INFO:teuthology.orchestra.run.smithi181.stdout: auth supported = cephx 2024-09-11T14:40:40.728 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:40.728 INFO:teuthology.orchestra.run.smithi181.stdout: filestore xattr use omap = true 2024-09-11T14:40:40.728 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:40.728 INFO:teuthology.orchestra.run.smithi181.stdout: mon clock drift allowed = 1.000 2024-09-11T14:40:40.728 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:40.728 INFO:teuthology.orchestra.run.smithi181.stdout: osd crush chooseleaf type = 0 2024-09-11T14:40:40.728 INFO:teuthology.orchestra.run.smithi181.stdout: auth debug = true 2024-09-11T14:40:40.728 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:40.729 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on old message = true 2024-09-11T14:40:40.729 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on bug = true 2024-09-11T14:40:40.729 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:40.729 INFO:teuthology.orchestra.run.smithi181.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T14:40:40.729 INFO:teuthology.orchestra.run.smithi181.stdout: mon pg warn max object skew = 0 2024-09-11T14:40:40.729 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:40.729 INFO:teuthology.orchestra.run.smithi181.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T14:40:40.729 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T14:40:40.729 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:40.729 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default size = 2 2024-09-11T14:40:40.729 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:40.729 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow primary affinity = true 2024-09-11T14:40:40.729 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow pg remap = true 2024-09-11T14:40:40.730 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on legacy crush tunables = false 2024-09-11T14:40:40.730 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on crush straw calc version zero = false 2024-09-11T14:40:40.730 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on no sortbitwise = false 2024-09-11T14:40:40.730 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on osd down out interval zero = false 2024-09-11T14:40:40.730 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on too few osds = false 2024-09-11T14:40:40.730 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T14:40:40.730 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T14:40:40.730 INFO:teuthology.orchestra.run.smithi181.stdout: mon_allow_pool_size_one = true 2024-09-11T14:40:40.730 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:40.730 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T14:40:40.730 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:40.730 INFO:teuthology.orchestra.run.smithi181.stdout: osd default data pool replay window = 5 2024-09-11T14:40:40.730 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:40.731 INFO:teuthology.orchestra.run.smithi181.stdout: mon allow pool delete = true 2024-09-11T14:40:40.731 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:40.731 INFO:teuthology.orchestra.run.smithi181.stdout: mon cluster log file level = debug 2024-09-11T14:40:40.731 INFO:teuthology.orchestra.run.smithi181.stdout: debug asserts on shutdown = true 2024-09-11T14:40:40.731 INFO:teuthology.orchestra.run.smithi181.stdout: mon health detail to clog = false 2024-09-11T14:40:40.731 INFO:teuthology.orchestra.run.smithi181.stdout: mon host = 172.21.15.181 2024-09-11T14:40:40.731 INFO:teuthology.orchestra.run.smithi181.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-09-11T14:40:40.731 INFO:teuthology.orchestra.run.smithi181.stdout: ms client mode = crc 2024-09-11T14:40:40.731 INFO:teuthology.orchestra.run.smithi181.stdout: ms cluster mode = crc 2024-09-11T14:40:40.731 INFO:teuthology.orchestra.run.smithi181.stdout: ms mon client mode = crc 2024-09-11T14:40:40.731 INFO:teuthology.orchestra.run.smithi181.stdout: ms mon cluster mode = crc 2024-09-11T14:40:40.731 INFO:teuthology.orchestra.run.smithi181.stdout: ms mon service mode = crc 2024-09-11T14:40:40.731 INFO:teuthology.orchestra.run.smithi181.stdout: ms service mode = crc 2024-09-11T14:40:40.732 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_min_size = 2 2024-09-11T14:40:40.732 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_size = 3 2024-09-11T14:40:40.732 INFO:teuthology.orchestra.run.smithi181.stdout: fsid = 0f7e90de-623e-43a2-a836-b493e06f03fc 2024-09-11T14:40:40.732 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:40.732 INFO:teuthology.orchestra.run.smithi181.stdout:[osd] 2024-09-11T14:40:40.732 INFO:teuthology.orchestra.run.smithi181.stdout: osd journal size = 100 2024-09-11T14:40:40.732 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:40.732 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub load threshold = 5.0 2024-09-11T14:40:40.732 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub max interval = 120 2024-09-11T14:40:40.732 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock profile = high_recovery_ops 2024-09-11T14:40:40.733 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:40.733 INFO:teuthology.orchestra.run.smithi181.stdout: osd recover clone overlap = true 2024-09-11T14:40:40.733 INFO:teuthology.orchestra.run.smithi181.stdout: osd recovery max chunk = 1048576 2024-09-11T14:40:40.733 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:40.733 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug shutdown = true 2024-09-11T14:40:40.733 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug op order = true 2024-09-11T14:40:40.733 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify stray on activate = true 2024-09-11T14:40:40.733 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug trim objects = true 2024-09-11T14:40:40.733 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:40.734 INFO:teuthology.orchestra.run.smithi181.stdout: osd open classes on start = true 2024-09-11T14:40:40.734 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug pg log writeout = true 2024-09-11T14:40:40.734 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:40.734 INFO:teuthology.orchestra.run.smithi181.stdout: osd deep scrub update digest min age = 30 2024-09-11T14:40:40.734 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:40.734 INFO:teuthology.orchestra.run.smithi181.stdout: osd map max advance = 10 2024-09-11T14:40:40.734 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:40.734 INFO:teuthology.orchestra.run.smithi181.stdout: journal zero on create = true 2024-09-11T14:40:40.734 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:40.734 INFO:teuthology.orchestra.run.smithi181.stdout: filestore ondisk finisher threads = 3 2024-09-11T14:40:40.734 INFO:teuthology.orchestra.run.smithi181.stdout: filestore apply finisher threads = 3 2024-09-11T14:40:40.734 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:40.734 INFO:teuthology.orchestra.run.smithi181.stdout: bdev debug aio = true 2024-09-11T14:40:40.734 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug misdirected ops = true 2024-09-11T14:40:40.735 INFO:teuthology.orchestra.run.smithi181.stdout: debug monc = 20 2024-09-11T14:40:40.735 INFO:teuthology.orchestra.run.smithi181.stdout: crimson alien thread cpu cores = 6-7 2024-09-11T14:40:40.735 INFO:teuthology.orchestra.run.smithi181.stdout: crimson osd obc lru size = 10 2024-09-11T14:40:40.735 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-11T14:40:40.735 INFO:teuthology.orchestra.run.smithi181.stdout: debug osd = 20 2024-09-11T14:40:40.735 INFO:teuthology.orchestra.run.smithi181.stdout: debug seastore = 20 2024-09-11T14:40:40.735 INFO:teuthology.orchestra.run.smithi181.stdout: debug seastore backref = 20 2024-09-11T14:40:40.735 INFO:teuthology.orchestra.run.smithi181.stdout: debug seastore cache = 20 2024-09-11T14:40:40.735 INFO:teuthology.orchestra.run.smithi181.stdout: debug seastore cleaner = 20 2024-09-11T14:40:40.735 INFO:teuthology.orchestra.run.smithi181.stdout: debug seastore device = 20 2024-09-11T14:40:40.735 INFO:teuthology.orchestra.run.smithi181.stdout: debug seastore epm = 20 2024-09-11T14:40:40.735 INFO:teuthology.orchestra.run.smithi181.stdout: debug seastore fixedkv tree = 20 2024-09-11T14:40:40.735 INFO:teuthology.orchestra.run.smithi181.stdout: debug seastore journal = 20 2024-09-11T14:40:40.735 INFO:teuthology.orchestra.run.smithi181.stdout: debug seastore lba = 20 2024-09-11T14:40:40.735 INFO:teuthology.orchestra.run.smithi181.stdout: debug seastore odata = 20 2024-09-11T14:40:40.736 INFO:teuthology.orchestra.run.smithi181.stdout: debug seastore ompap = 20 2024-09-11T14:40:40.736 INFO:teuthology.orchestra.run.smithi181.stdout: debug seastore onode = 20 2024-09-11T14:40:40.736 INFO:teuthology.orchestra.run.smithi181.stdout: debug seastore t = 20 2024-09-11T14:40:40.736 INFO:teuthology.orchestra.run.smithi181.stdout: debug seastore tm = 20 2024-09-11T14:40:40.736 INFO:teuthology.orchestra.run.smithi181.stdout: osd blocked scrub grace period = 3600 2024-09-11T14:40:40.736 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug reject backfill probability = 0.3 2024-09-11T14:40:40.736 INFO:teuthology.orchestra.run.smithi181.stdout: osd delete sleep = 1 2024-09-11T14:40:40.736 INFO:teuthology.orchestra.run.smithi181.stdout: osd max backfills = 3 2024-09-11T14:40:40.736 INFO:teuthology.orchestra.run.smithi181.stdout: osd max markdown count = 1000 2024-09-11T14:40:40.736 INFO:teuthology.orchestra.run.smithi181.stdout: osd objectstore = seastore 2024-09-11T14:40:40.736 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub min interval = 60 2024-09-11T14:40:40.736 INFO:teuthology.orchestra.run.smithi181.stdout: osd shutdown pgref assert = True 2024-09-11T14:40:40.736 INFO:teuthology.orchestra.run.smithi181.stdout: osd snap trim sleep = 2 2024-09-11T14:40:40.736 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:40.737 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr] 2024-09-11T14:40:40.737 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-11T14:40:40.737 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgr = 20 2024-09-11T14:40:40.737 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-11T14:40:40.737 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-11T14:40:40.737 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-11T14:40:40.737 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-11T14:40:40.738 INFO:teuthology.orchestra.run.smithi181.stdout: mgr/telemetry/nag = false 2024-09-11T14:40:40.738 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:40.738 INFO:teuthology.orchestra.run.smithi181.stdout:[mon] 2024-09-11T14:40:40.738 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-11T14:40:40.738 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-11T14:40:40.738 INFO:teuthology.orchestra.run.smithi181.stdout: debug paxos = 20 2024-09-11T14:40:40.738 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-11T14:40:40.738 INFO:teuthology.orchestra.run.smithi181.stdout: mon data avail warn = 5 2024-09-11T14:40:40.738 INFO:teuthology.orchestra.run.smithi181.stdout: mon mgr mkfs grace = 240 2024-09-11T14:40:40.738 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-11T14:40:40.738 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd reporter subtree level = osd 2024-09-11T14:40:40.738 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd prime pg temp = true 2024-09-11T14:40:40.739 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-11T14:40:40.739 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:40.739 INFO:teuthology.orchestra.run.smithi181.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T14:40:40.739 INFO:teuthology.orchestra.run.smithi181.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T14:40:40.739 INFO:teuthology.orchestra.run.smithi181.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T14:40:40.739 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:40.739 INFO:teuthology.orchestra.run.smithi181.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T14:40:40.739 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T14:40:40.739 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T14:40:40.739 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:40.739 INFO:teuthology.orchestra.run.smithi181.stdout: # 1m isn't quite enough 2024-09-11T14:40:40.739 INFO:teuthology.orchestra.run.smithi181.stdout: mon_down_mkfs_grace = 2m 2024-09-11T14:40:40.739 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:40.739 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_filestore_osds = false 2024-09-11T14:40:40.740 INFO:teuthology.orchestra.run.smithi181.stdout: mon min osdmap epochs = 50 2024-09-11T14:40:40.740 INFO:teuthology.orchestra.run.smithi181.stdout: mon osdmap full prune interval = 2 2024-09-11T14:40:40.740 INFO:teuthology.orchestra.run.smithi181.stdout: mon osdmap full prune min = 15 2024-09-11T14:40:40.740 INFO:teuthology.orchestra.run.smithi181.stdout: mon osdmap full prune txsize = 2 2024-09-11T14:40:40.740 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default crimson = True 2024-09-11T14:40:40.740 INFO:teuthology.orchestra.run.smithi181.stdout: paxos service trim min = 10 2024-09-11T14:40:40.740 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:40:40.740 INFO:teuthology.orchestra.run.smithi181.stdout:[client] 2024-09-11T14:40:40.740 INFO:teuthology.orchestra.run.smithi181.stdout: rgw cache enabled = true 2024-09-11T14:40:40.740 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable ops log = true 2024-09-11T14:40:40.740 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable usage log = true 2024-09-11T14:40:40.740 INFO:teuthology.orchestra.run.smithi181.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T14:40:40.740 INFO:teuthology.orchestra.run.smithi181.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T14:40:40.740 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.a] 2024-09-11T14:40:40.740 INFO:teuthology.orchestra.run.smithi181.stdout:[client.0] 2024-09-11T14:40:40.741 INFO:teuthology.orchestra.run.smithi181.stdout: admin socket = /var/run/ceph/ceph-$name.asok 2024-09-11T14:40:40.741 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.0] 2024-09-11T14:40:40.741 INFO:teuthology.orchestra.run.smithi181.stdout: crimson seastar cpu cores = 0-2 2024-09-11T14:40:40.741 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.1] 2024-09-11T14:40:40.741 INFO:teuthology.orchestra.run.smithi181.stdout: crimson seastar cpu cores = 3-5 2024-09-11T14:40:40.741 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.2] 2024-09-11T14:40:40.741 INFO:teuthology.orchestra.run.smithi181.stdout: crimson seastar cpu cores = 0-2 2024-09-11T14:40:40.741 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.3] 2024-09-11T14:40:40.741 INFO:teuthology.orchestra.run.smithi181.stdout: crimson seastar cpu cores = 3-5 2024-09-11T14:40:40.742 INFO:teuthology.orchestra.run.smithi203.stdout:[global] 2024-09-11T14:40:40.742 INFO:teuthology.orchestra.run.smithi203.stdout: chdir = "" 2024-09-11T14:40:40.742 INFO:teuthology.orchestra.run.smithi203.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T14:40:40.742 INFO:teuthology.orchestra.run.smithi203.stdout: auth supported = cephx 2024-09-11T14:40:40.742 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:40.742 INFO:teuthology.orchestra.run.smithi203.stdout: filestore xattr use omap = true 2024-09-11T14:40:40.742 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:40.742 INFO:teuthology.orchestra.run.smithi203.stdout: mon clock drift allowed = 1.000 2024-09-11T14:40:40.742 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:40.742 INFO:teuthology.orchestra.run.smithi203.stdout: osd crush chooseleaf type = 0 2024-09-11T14:40:40.742 INFO:teuthology.orchestra.run.smithi203.stdout: auth debug = true 2024-09-11T14:40:40.742 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:40.742 INFO:teuthology.orchestra.run.smithi203.stdout: ms die on old message = true 2024-09-11T14:40:40.743 INFO:teuthology.orchestra.run.smithi203.stdout: ms die on bug = true 2024-09-11T14:40:40.743 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:40.743 INFO:teuthology.orchestra.run.smithi203.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T14:40:40.743 INFO:teuthology.orchestra.run.smithi203.stdout: mon pg warn max object skew = 0 2024-09-11T14:40:40.743 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:40.743 INFO:teuthology.orchestra.run.smithi203.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T14:40:40.743 INFO:teuthology.orchestra.run.smithi203.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T14:40:40.743 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:40.743 INFO:teuthology.orchestra.run.smithi203.stdout: osd pool default size = 2 2024-09-11T14:40:40.743 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:40.743 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd allow primary affinity = true 2024-09-11T14:40:40.743 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd allow pg remap = true 2024-09-11T14:40:40.743 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on legacy crush tunables = false 2024-09-11T14:40:40.743 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on crush straw calc version zero = false 2024-09-11T14:40:40.743 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on no sortbitwise = false 2024-09-11T14:40:40.743 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on osd down out interval zero = false 2024-09-11T14:40:40.744 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on too few osds = false 2024-09-11T14:40:40.744 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T14:40:40.744 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T14:40:40.744 INFO:teuthology.orchestra.run.smithi203.stdout: mon_allow_pool_size_one = true 2024-09-11T14:40:40.744 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:40.744 INFO:teuthology.orchestra.run.smithi203.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T14:40:40.744 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:40.744 INFO:teuthology.orchestra.run.smithi203.stdout: osd default data pool replay window = 5 2024-09-11T14:40:40.744 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:40.744 INFO:teuthology.orchestra.run.smithi203.stdout: mon allow pool delete = true 2024-09-11T14:40:40.744 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:40.744 INFO:teuthology.orchestra.run.smithi203.stdout: mon cluster log file level = debug 2024-09-11T14:40:40.744 INFO:teuthology.orchestra.run.smithi203.stdout: debug asserts on shutdown = true 2024-09-11T14:40:40.744 INFO:teuthology.orchestra.run.smithi203.stdout: mon health detail to clog = false 2024-09-11T14:40:40.745 INFO:teuthology.orchestra.run.smithi203.stdout: mon host = 172.21.15.181 2024-09-11T14:40:40.745 INFO:teuthology.orchestra.run.smithi203.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-09-11T14:40:40.745 INFO:teuthology.orchestra.run.smithi203.stdout: ms client mode = crc 2024-09-11T14:40:40.745 INFO:teuthology.orchestra.run.smithi203.stdout: ms cluster mode = crc 2024-09-11T14:40:40.745 INFO:teuthology.orchestra.run.smithi203.stdout: ms mon client mode = crc 2024-09-11T14:40:40.745 INFO:teuthology.orchestra.run.smithi203.stdout: ms mon cluster mode = crc 2024-09-11T14:40:40.745 INFO:teuthology.orchestra.run.smithi203.stdout: ms mon service mode = crc 2024-09-11T14:40:40.745 INFO:teuthology.orchestra.run.smithi203.stdout: ms service mode = crc 2024-09-11T14:40:40.745 INFO:teuthology.orchestra.run.smithi203.stdout: osd_pool_default_min_size = 2 2024-09-11T14:40:40.745 INFO:teuthology.orchestra.run.smithi203.stdout: osd_pool_default_size = 3 2024-09-11T14:40:40.745 INFO:teuthology.orchestra.run.smithi203.stdout: fsid = 0f7e90de-623e-43a2-a836-b493e06f03fc 2024-09-11T14:40:40.745 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:40.745 INFO:teuthology.orchestra.run.smithi203.stdout:[osd] 2024-09-11T14:40:40.745 INFO:teuthology.orchestra.run.smithi203.stdout: osd journal size = 100 2024-09-11T14:40:40.746 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:40.746 INFO:teuthology.orchestra.run.smithi203.stdout: osd scrub load threshold = 5.0 2024-09-11T14:40:40.746 INFO:teuthology.orchestra.run.smithi203.stdout: osd scrub max interval = 120 2024-09-11T14:40:40.746 INFO:teuthology.orchestra.run.smithi203.stdout: osd mclock profile = high_recovery_ops 2024-09-11T14:40:40.746 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:40.746 INFO:teuthology.orchestra.run.smithi203.stdout: osd recover clone overlap = true 2024-09-11T14:40:40.746 INFO:teuthology.orchestra.run.smithi203.stdout: osd recovery max chunk = 1048576 2024-09-11T14:40:40.746 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:40.746 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug shutdown = true 2024-09-11T14:40:40.746 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug op order = true 2024-09-11T14:40:40.746 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug verify stray on activate = true 2024-09-11T14:40:40.746 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug trim objects = true 2024-09-11T14:40:40.746 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:40.746 INFO:teuthology.orchestra.run.smithi203.stdout: osd open classes on start = true 2024-09-11T14:40:40.747 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug pg log writeout = true 2024-09-11T14:40:40.747 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:40.747 INFO:teuthology.orchestra.run.smithi203.stdout: osd deep scrub update digest min age = 30 2024-09-11T14:40:40.747 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:40.747 INFO:teuthology.orchestra.run.smithi203.stdout: osd map max advance = 10 2024-09-11T14:40:40.747 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:40.747 INFO:teuthology.orchestra.run.smithi203.stdout: journal zero on create = true 2024-09-11T14:40:40.747 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:40.747 INFO:teuthology.orchestra.run.smithi203.stdout: filestore ondisk finisher threads = 3 2024-09-11T14:40:40.747 INFO:teuthology.orchestra.run.smithi203.stdout: filestore apply finisher threads = 3 2024-09-11T14:40:40.747 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:40.747 INFO:teuthology.orchestra.run.smithi203.stdout: bdev debug aio = true 2024-09-11T14:40:40.747 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug misdirected ops = true 2024-09-11T14:40:40.747 INFO:teuthology.orchestra.run.smithi203.stdout: debug monc = 20 2024-09-11T14:40:40.748 INFO:teuthology.orchestra.run.smithi203.stdout: crimson alien thread cpu cores = 6-7 2024-09-11T14:40:40.748 INFO:teuthology.orchestra.run.smithi203.stdout: crimson osd obc lru size = 10 2024-09-11T14:40:40.748 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-09-11T14:40:40.748 INFO:teuthology.orchestra.run.smithi203.stdout: debug osd = 20 2024-09-11T14:40:40.748 INFO:teuthology.orchestra.run.smithi203.stdout: debug seastore = 20 2024-09-11T14:40:40.748 INFO:teuthology.orchestra.run.smithi203.stdout: debug seastore backref = 20 2024-09-11T14:40:40.748 INFO:teuthology.orchestra.run.smithi203.stdout: debug seastore cache = 20 2024-09-11T14:40:40.748 INFO:teuthology.orchestra.run.smithi203.stdout: debug seastore cleaner = 20 2024-09-11T14:40:40.748 INFO:teuthology.orchestra.run.smithi203.stdout: debug seastore device = 20 2024-09-11T14:40:40.748 INFO:teuthology.orchestra.run.smithi203.stdout: debug seastore epm = 20 2024-09-11T14:40:40.748 INFO:teuthology.orchestra.run.smithi203.stdout: debug seastore fixedkv tree = 20 2024-09-11T14:40:40.748 INFO:teuthology.orchestra.run.smithi203.stdout: debug seastore journal = 20 2024-09-11T14:40:40.748 INFO:teuthology.orchestra.run.smithi203.stdout: debug seastore lba = 20 2024-09-11T14:40:40.748 INFO:teuthology.orchestra.run.smithi203.stdout: debug seastore odata = 20 2024-09-11T14:40:40.749 INFO:teuthology.orchestra.run.smithi203.stdout: debug seastore ompap = 20 2024-09-11T14:40:40.749 INFO:teuthology.orchestra.run.smithi203.stdout: debug seastore onode = 20 2024-09-11T14:40:40.749 INFO:teuthology.orchestra.run.smithi203.stdout: debug seastore t = 20 2024-09-11T14:40:40.749 INFO:teuthology.orchestra.run.smithi203.stdout: debug seastore tm = 20 2024-09-11T14:40:40.749 INFO:teuthology.orchestra.run.smithi203.stdout: osd blocked scrub grace period = 3600 2024-09-11T14:40:40.749 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug reject backfill probability = 0.3 2024-09-11T14:40:40.749 INFO:teuthology.orchestra.run.smithi203.stdout: osd delete sleep = 1 2024-09-11T14:40:40.749 INFO:teuthology.orchestra.run.smithi203.stdout: osd max backfills = 3 2024-09-11T14:40:40.749 INFO:teuthology.orchestra.run.smithi203.stdout: osd max markdown count = 1000 2024-09-11T14:40:40.749 INFO:teuthology.orchestra.run.smithi203.stdout: osd objectstore = seastore 2024-09-11T14:40:40.749 INFO:teuthology.orchestra.run.smithi203.stdout: osd scrub min interval = 60 2024-09-11T14:40:40.749 INFO:teuthology.orchestra.run.smithi203.stdout: osd shutdown pgref assert = True 2024-09-11T14:40:40.749 INFO:teuthology.orchestra.run.smithi203.stdout: osd snap trim sleep = 2 2024-09-11T14:40:40.750 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:40.750 INFO:teuthology.orchestra.run.smithi203.stdout:[mgr] 2024-09-11T14:40:40.750 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-09-11T14:40:40.750 INFO:teuthology.orchestra.run.smithi203.stdout: debug mgr = 20 2024-09-11T14:40:40.750 INFO:teuthology.orchestra.run.smithi203.stdout: debug mon = 20 2024-09-11T14:40:40.750 INFO:teuthology.orchestra.run.smithi203.stdout: debug auth = 20 2024-09-11T14:40:40.750 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min pgs per osd = 4 2024-09-11T14:40:40.750 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min bytes per osd = 10 2024-09-11T14:40:40.750 INFO:teuthology.orchestra.run.smithi203.stdout: mgr/telemetry/nag = false 2024-09-11T14:40:40.750 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:40.750 INFO:teuthology.orchestra.run.smithi203.stdout:[mon] 2024-09-11T14:40:40.750 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-09-11T14:40:40.750 INFO:teuthology.orchestra.run.smithi203.stdout: debug mon = 20 2024-09-11T14:40:40.750 INFO:teuthology.orchestra.run.smithi203.stdout: debug paxos = 20 2024-09-11T14:40:40.750 INFO:teuthology.orchestra.run.smithi203.stdout: debug auth = 20 2024-09-11T14:40:40.751 INFO:teuthology.orchestra.run.smithi203.stdout: mon data avail warn = 5 2024-09-11T14:40:40.751 INFO:teuthology.orchestra.run.smithi203.stdout: mon mgr mkfs grace = 240 2024-09-11T14:40:40.751 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min pgs per osd = 4 2024-09-11T14:40:40.751 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd reporter subtree level = osd 2024-09-11T14:40:40.751 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd prime pg temp = true 2024-09-11T14:40:40.751 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min bytes per osd = 10 2024-09-11T14:40:40.751 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:40.751 INFO:teuthology.orchestra.run.smithi203.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T14:40:40.751 INFO:teuthology.orchestra.run.smithi203.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T14:40:40.751 INFO:teuthology.orchestra.run.smithi203.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T14:40:40.751 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:40.751 INFO:teuthology.orchestra.run.smithi203.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T14:40:40.751 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T14:40:40.751 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T14:40:40.751 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:40.751 INFO:teuthology.orchestra.run.smithi203.stdout: # 1m isn't quite enough 2024-09-11T14:40:40.752 INFO:teuthology.orchestra.run.smithi203.stdout: mon_down_mkfs_grace = 2m 2024-09-11T14:40:40.752 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:40.752 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_filestore_osds = false 2024-09-11T14:40:40.752 INFO:teuthology.orchestra.run.smithi203.stdout: mon min osdmap epochs = 50 2024-09-11T14:40:40.752 INFO:teuthology.orchestra.run.smithi203.stdout: mon osdmap full prune interval = 2 2024-09-11T14:40:40.752 INFO:teuthology.orchestra.run.smithi203.stdout: mon osdmap full prune min = 15 2024-09-11T14:40:40.752 INFO:teuthology.orchestra.run.smithi203.stdout: mon osdmap full prune txsize = 2 2024-09-11T14:40:40.752 INFO:teuthology.orchestra.run.smithi203.stdout: osd pool default crimson = True 2024-09-11T14:40:40.752 INFO:teuthology.orchestra.run.smithi203.stdout: paxos service trim min = 10 2024-09-11T14:40:40.752 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-11T14:40:40.752 INFO:teuthology.orchestra.run.smithi203.stdout:[client] 2024-09-11T14:40:40.752 INFO:teuthology.orchestra.run.smithi203.stdout: rgw cache enabled = true 2024-09-11T14:40:40.752 INFO:teuthology.orchestra.run.smithi203.stdout: rgw enable ops log = true 2024-09-11T14:40:40.752 INFO:teuthology.orchestra.run.smithi203.stdout: rgw enable usage log = true 2024-09-11T14:40:40.752 INFO:teuthology.orchestra.run.smithi203.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T14:40:40.753 INFO:teuthology.orchestra.run.smithi203.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T14:40:40.753 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.a] 2024-09-11T14:40:40.753 INFO:teuthology.orchestra.run.smithi203.stdout:[client.0] 2024-09-11T14:40:40.753 INFO:teuthology.orchestra.run.smithi203.stdout: admin socket = /var/run/ceph/ceph-$name.asok 2024-09-11T14:40:40.753 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.0] 2024-09-11T14:40:40.753 INFO:teuthology.orchestra.run.smithi203.stdout: crimson seastar cpu cores = 0-2 2024-09-11T14:40:40.753 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.1] 2024-09-11T14:40:40.753 INFO:teuthology.orchestra.run.smithi203.stdout: crimson seastar cpu cores = 3-5 2024-09-11T14:40:40.753 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.2] 2024-09-11T14:40:40.753 INFO:teuthology.orchestra.run.smithi203.stdout: crimson seastar cpu cores = 0-2 2024-09-11T14:40:40.753 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.3] 2024-09-11T14:40:40.753 INFO:teuthology.orchestra.run.smithi203.stdout: crimson seastar cpu cores = 3-5 2024-09-11T14:40:40.755 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-11T14:40:40.755 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-11T14:40:40.860 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-11T14:40:40.861 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-11T14:40:40.861 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-11T14:40:40.876 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-11T14:40:40.877 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-11T14:40:40.931 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi181.front.sepia.ceph.com 2024-09-11T14:40:40.931 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-11T14:40:40.931 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-11T14:40:40.931 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T14:40:41.005 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-11T14:40:41.005 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:40:41.060 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi203.front.sepia.ceph.com 2024-09-11T14:40:41.061 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-11T14:40:41.061 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-11T14:40:41.061 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T14:40:41.096 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-11T14:40:41.097 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:40:41.152 INFO:tasks.ceph:Setting up mon nodes... 2024-09-11T14:40:41.152 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-11T14:40:41.153 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-11T14:40:41.271 INFO:teuthology.orchestra.run.smithi203.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-11T14:40:41.274 INFO:tasks.ceph:Setting up mds nodes... 2024-09-11T14:40:41.274 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-11T14:40:41.274 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-11T14:40:41.324 INFO:teuthology.orchestra.run.smithi181.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-11T14:40:41.337 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-11T14:40:41.387 INFO:teuthology.orchestra.run.smithi203.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-09-11T14:40:41.400 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-11T14:40:41.401 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi181.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi203.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-09-11T14:40:41.401 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-11T14:40:41.427 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-09-11T14:40:41.427 INFO:tasks.ceph:role: osd.0 2024-09-11T14:40:41.427 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-11T14:40:41.428 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-11T14:40:41.499 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T14:40:41.500 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T14:40:41.500 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T14:40:41.500 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T14:40:41.500 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T14:40:41.500 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-09-11T14:40:41.500 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T14:40:41.500 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T14:40:41.500 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T14:40:41.500 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T14:40:42.020 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-09-11T14:40:42.022 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-09-11T14:40:42.022 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-11T14:40:42.166 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-11T14:40:42.198 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-11T14:40:42.263 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-09-11T14:40:42.263 INFO:tasks.ceph:role: osd.1 2024-09-11T14:40:42.263 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-11T14:40:42.264 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-11T14:40:42.335 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T14:40:42.335 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T14:40:42.335 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T14:40:42.336 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T14:40:42.336 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T14:40:42.336 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-09-11T14:40:42.336 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T14:40:42.336 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T14:40:42.336 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T14:40:42.336 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T14:40:42.782 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-09-11T14:40:42.784 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-09-11T14:40:42.785 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-11T14:40:42.877 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-11T14:40:42.905 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:40:43.205 INFO:teuthology.orchestra.run.smithi181.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:40:43.206 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-09-11 14:40:43,206 seastar - Reactor backend: linux-aio 2024-09-11T14:40:43.218 INFO:teuthology.orchestra.run.smithi181.stderr:==27898==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:40:43.342 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-09-11 14:40:43,342 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-11T14:40:43.342 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-09-11 14:40:43,342 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 0-2 2024-09-11T14:40:43.343 INFO:teuthology.orchestra.run.smithi181.stderr:DEBUG 2024-09-11 14:40:43,343 [shard 0:main] seastar - reactor::drain 2024-09-11T14:40:43.882 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-09-11T14:40:43.882 INFO:teuthology.orchestra.run.smithi181.stderr:Suppressions used: 2024-09-11T14:40:43.882 INFO:teuthology.orchestra.run.smithi181.stderr: count bytes template 2024-09-11T14:40:43.882 INFO:teuthology.orchestra.run.smithi181.stderr: 1 8 MallocExtension::Register 2024-09-11T14:40:43.882 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-09-11T14:40:43.882 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-11T14:40:43.939 INFO:teuthology.orchestra.run.smithi181.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:40:43.943 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-09-11 14:40:43,942 seastar - Reactor backend: linux-aio 2024-09-11T14:40:43.953 INFO:teuthology.orchestra.run.smithi181.stderr:==27894==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:40:44.245 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x3628461 0x36e82eb 0xba29217 0xba29498 0xba2964e 0xba2979b 0xb9d1e3e 0xb9d64f1 0x984ae51 0x9b01034 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:44.245 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-11T14:40:44.245 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x3780550 0x3b92062 0x3b92268 0xb9f075a 0xb9f1bfa 0xb9f1e74 0xb9f201c 0xb9f3dd7 0xb9f3fd3 0xba38ebd 0xb9e038f 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:44.245 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-11T14:40:44.302 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x3ad43 0x10d76c 0xdd31a 0x4b14c 0x3b920e7 0x3b92268 0xba052ce 0xba407e3 0xba40b6e 0xb9e0f68 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b066a9 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:44.303 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-11T14:40:44.303 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 90 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0xbbf9d7c 0xc4b9022 0xc4b9862 0xb0da5b9 0xb0d4c90 0xac9c741 0xad02411 0xaccf6d6 0x994da0a 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:44.303 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-11T14:40:44.303 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x133c2d 0xdd332 0x4ab1c 0xb9f07c3 0xb9f1bfa 0xb9f1e74 0xb9f201c 0xb9f3dd7 0xb9f4222 0xb9e014c 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:40:44.303 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-11T14:40:44.361 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 123 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x6114c7a 0xba3c37a 0xba3c71e 0xba3c930 0xb9d97af 0xb9d9bd4 0xb9dfb97 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:40:44.361 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-11T14:40:44.494 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x4ab17 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:40:44.494 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-11T14:40:44.551 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x5549385 0x9e3b019 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:40:44.552 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-11T14:40:44.586 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x4ac00 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:44.586 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-11T14:40:44.589 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x5549131 0x554a11c 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:44.589 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-11T14:40:44.644 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 122 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x4abc2 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:44.644 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-11T14:40:44.646 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x9e3aff4 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:44.646 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-11T14:40:44.961 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x360e4c5 0x3b92070 0x3b92268 0xb9f0927 0xb9f1bfa 0xb9f1e74 0xb9f201c 0xb9f45e7 0xb9e014c 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93195d6 0x931a07a 0x931a0fd 0x931a182 0x931a284 0x931a37c 0x931a402 0x931a4bb 0x931a671 0x931a7a9 0x931a824 0x931a8a9 0x931aa04 0x931ab1e 0x931aba3 0x931ac81 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:44.961 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-11T14:40:44.961 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x3acdab6 0x3b9225d 0xb9f0927 0xb9f1bfa 0xb9f1e74 0xb9f201c 0xb9f3dd7 0xb9f3fd3 0xba38ebd 0xb9e038f 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93195d6 0x931a07a 0x931a0fd 0x931a182 0x931a284 0x931a37c 0x931a402 0x931a4bb 0x931a671 0x931a7a9 0x931a824 0x931a8a9 0x931aa04 0x931ab1e 0x931aba3 0x931ac81 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:44.962 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-11T14:40:45.120 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x4ab9d 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:40:45.120 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-11T14:40:46.043 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-09-11T14:40:46.043 INFO:teuthology.orchestra.run.smithi181.stderr:Suppressions used: 2024-09-11T14:40:46.043 INFO:teuthology.orchestra.run.smithi181.stderr: count bytes template 2024-09-11T14:40:46.043 INFO:teuthology.orchestra.run.smithi181.stderr: 1 8 MallocExtension::Register 2024-09-11T14:40:46.043 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-09-11T14:40:46.043 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-11T14:40:46.072 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-11T14:40:46.097 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:40:46.376 INFO:teuthology.orchestra.run.smithi181.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:40:46.377 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-09-11 14:40:46,377 seastar - Reactor backend: linux-aio 2024-09-11T14:40:46.382 INFO:teuthology.orchestra.run.smithi181.stderr:==27946==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:40:46.505 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-09-11 14:40:46,505 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-11T14:40:46.505 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-09-11 14:40:46,505 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 3-5 2024-09-11T14:40:46.506 INFO:teuthology.orchestra.run.smithi181.stderr:DEBUG 2024-09-11 14:40:46,506 [shard 0:main] seastar - reactor::drain 2024-09-11T14:40:47.007 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-09-11T14:40:47.007 INFO:teuthology.orchestra.run.smithi181.stderr:Suppressions used: 2024-09-11T14:40:47.007 INFO:teuthology.orchestra.run.smithi181.stderr: count bytes template 2024-09-11T14:40:47.007 INFO:teuthology.orchestra.run.smithi181.stderr: 1 8 MallocExtension::Register 2024-09-11T14:40:47.007 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-09-11T14:40:47.007 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-11T14:40:47.067 INFO:teuthology.orchestra.run.smithi181.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:40:47.071 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-09-11 14:40:47,070 seastar - Reactor backend: linux-aio 2024-09-11T14:40:47.089 INFO:teuthology.orchestra.run.smithi181.stderr:==27944==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:40:47.372 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x10a2ba 0xfb55e 0x376ee9b 0xb9d9722 0xb9d9bd4 0xb9dfb97 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:47.373 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-11T14:40:47.373 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x8b581 0x10d76c 0x42124 0x3ef7b 0xfd531 0xb85ab4d 0xb85ac00 0xb85ad16 0xb85adf6 0xb85b4b6 0xb85b12f 0xb85b621 0xb85ba2c 0xb85bceb 0xba2da49 0xba312a0 0xb9e403b 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:47.373 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-11T14:40:47.430 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x119673 0x11d39b 0x41608 0x41bbe 0x41c92 0x41f68 0x4214e 0x3ef7b 0xfd531 0x3c3af50 0x556073c 0x9ed85bd 0x9ed8b76 0x9ed8f1e 0x9edc2f8 0x9edcf54 0x9edd217 0x9edde8e 0x9e293f2 0x9a07671 0x9a07c77 0x98fcbb0 0x93358b1 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:47.430 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffa813f9b5 0xffffffffa7f6ccd2 0xffffffffa88cd45f 0xffffffffa8a00130 2024-09-11T14:40:47.430 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 90 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x4aba4 0x6f7e38d 0x9ed847e 0x9ed8b76 0x9ed8f1e 0x9edc2f8 0x9edcf54 0x9edd217 0x9edde8e 0x9e293f2 0x9a07671 0x9a07c77 0x98fcbb0 0x93358b1 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:47.430 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-11T14:40:47.431 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0xb85a91c 0xb85ac23 0xb85ad16 0xb85adf6 0xb85b4b6 0xb85b621 0xb85ba2c 0xb85bceb 0xba2da49 0xba312a0 0xb9e403b 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:40:47.431 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-11T14:40:47.488 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 123 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0xb9d62bb 0x984ae51 0x9b01034 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:40:47.488 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-11T14:40:47.621 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x156e3 0xdd312 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:40:47.621 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-11T14:40:47.679 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x5549659 0x9e3b02e 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:40:47.679 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-11T14:40:47.724 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x4abc2 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:47.724 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-11T14:40:47.724 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x8b581 0x10d76c 0xdd31a 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:47.724 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-11T14:40:47.782 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x5549a31 0x9e3b03b 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:47.782 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-11T14:40:47.782 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 122 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x133c3f 0xdd332 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:47.782 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-11T14:40:48.088 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x959fff0 0x9b019a8 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93195d6 0x931a07a 0x931a0fd 0x931a182 0x931a284 0x931a37c 0x931a402 0x931a4bb 0x931a671 0x931a7a9 0x931a824 0x931a8a9 0x931aa04 0x931ab1e 0x931aba3 0x931ac81 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:48.088 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-11T14:40:48.088 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x3acd9f2 0x3b9225d 0xb9f0927 0xb9f1bfa 0xb9f1e74 0xb9f201c 0xb9f3dd7 0xb9f4222 0xb9e014c 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93195d6 0x931a07a 0x931a0fd 0x931a182 0x931a284 0x931a37c 0x931a402 0x931a4bb 0x931a671 0x931a7a9 0x931a824 0x931a8a9 0x931aa04 0x931ab1e 0x931aba3 0x931ac81 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:48.088 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-11T14:40:48.246 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x9e3b026 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:40:48.247 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-11T14:40:49.158 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-09-11T14:40:49.158 INFO:teuthology.orchestra.run.smithi181.stderr:Suppressions used: 2024-09-11T14:40:49.159 INFO:teuthology.orchestra.run.smithi181.stderr: count bytes template 2024-09-11T14:40:49.159 INFO:teuthology.orchestra.run.smithi181.stderr: 1 8 MallocExtension::Register 2024-09-11T14:40:49.159 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-09-11T14:40:49.159 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-11T14:40:49.184 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-11T14:40:49.209 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-11T14:40:49.238 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-09-11T14:40:49.239 INFO:tasks.ceph:role: osd.2 2024-09-11T14:40:49.239 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi203.front.sepia.ceph.com 2024-09-11T14:40:49.239 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-11T14:40:49.315 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T14:40:49.315 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T14:40:49.315 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T14:40:49.315 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T14:40:49.315 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T14:40:49.315 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-09-11T14:40:49.315 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T14:40:49.316 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T14:40:49.316 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T14:40:49.316 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T14:40:49.826 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-09-11T14:40:49.829 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-09-11T14:40:49.829 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-09-11T14:40:49.978 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-11T14:40:50.011 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-11T14:40:50.077 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-09-11T14:40:50.077 INFO:tasks.ceph:role: osd.3 2024-09-11T14:40:50.077 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi203.front.sepia.ceph.com 2024-09-11T14:40:50.077 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-11T14:40:50.150 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T14:40:50.150 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T14:40:50.150 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T14:40:50.151 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T14:40:50.151 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T14:40:50.151 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-09-11T14:40:50.151 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T14:40:50.151 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T14:40:50.151 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T14:40:50.151 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T14:40:50.602 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-09-11T14:40:50.604 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-09-11T14:40:50.605 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-3 2024-09-11T14:40:50.698 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-11T14:40:50.727 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:40:51.057 INFO:teuthology.orchestra.run.smithi203.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:40:51.058 INFO:teuthology.orchestra.run.smithi203.stderr:INFO 2024-09-11 14:40:51,057 seastar - Reactor backend: linux-aio 2024-09-11T14:40:51.077 INFO:teuthology.orchestra.run.smithi203.stderr:==27369==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:40:51.203 INFO:teuthology.orchestra.run.smithi203.stderr:INFO 2024-09-11 14:40:51,202 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-11T14:40:51.203 INFO:teuthology.orchestra.run.smithi203.stderr:INFO 2024-09-11 14:40:51,202 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 0-2 2024-09-11T14:40:51.204 INFO:teuthology.orchestra.run.smithi203.stderr:DEBUG 2024-09-11 14:40:51,203 [shard 0:main] seastar - reactor::drain 2024-09-11T14:40:51.752 INFO:teuthology.orchestra.run.smithi203.stderr:----------------------------------------------------- 2024-09-11T14:40:51.752 INFO:teuthology.orchestra.run.smithi203.stderr:Suppressions used: 2024-09-11T14:40:51.752 INFO:teuthology.orchestra.run.smithi203.stderr: count bytes template 2024-09-11T14:40:51.752 INFO:teuthology.orchestra.run.smithi203.stderr: 1 8 MallocExtension::Register 2024-09-11T14:40:51.752 INFO:teuthology.orchestra.run.smithi203.stderr:----------------------------------------------------- 2024-09-11T14:40:51.752 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-11T14:40:51.820 INFO:teuthology.orchestra.run.smithi203.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:40:51.824 INFO:teuthology.orchestra.run.smithi203.stderr:INFO 2024-09-11 14:40:51,823 seastar - Reactor backend: linux-aio 2024-09-11T14:40:51.833 INFO:teuthology.orchestra.run.smithi203.stderr:==27365==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:40:52.122 INFO:teuthology.orchestra.run.smithi203.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x4218a 0x3ecea 0xfc9b1 0x3870209 0xb85a732 0xb85a87a 0xb85a963 0xb85ac23 0xb85ad16 0xb85adf6 0xb85b076 0xb85b621 0xb85ba2c 0xb85bceb 0xba2da49 0xba2fa57 0xba2fcdf 0xba2fd8f 0xb9df75d 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:52.122 INFO:teuthology.orchestra.run.smithi203.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffa2e8c7a4 0xffffffffa3ad1d62 0xffffffffa3c00bb2 2024-09-11T14:40:52.122 INFO:teuthology.orchestra.run.smithi203.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x10bf45 0x10a29f 0xfca1b 0x3870209 0xba179c3 0xb9d61ef 0x984ae51 0x9b0161a 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:52.122 INFO:teuthology.orchestra.run.smithi203.stderr:kernel callstack: 2024-09-11T14:40:52.179 INFO:teuthology.orchestra.run.smithi203.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0xb85baa3 0xb85bceb 0xba2da49 0xba2fa57 0xba2fcdf 0xba2fd8f 0xb9df75d 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9ab9cb7 0x9b05946 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:52.179 INFO:teuthology.orchestra.run.smithi203.stderr:Reactor stalled for 90 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x3acda95 0x3b9225d 0xb9f768b 0xb9f7940 0xb9dfd11 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9ab9cb7 0x9b05946 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:52.179 INFO:teuthology.orchestra.run.smithi203.stderr:kernel callstack: 2024-09-11T14:40:52.179 INFO:teuthology.orchestra.run.smithi203.stderr:kernel callstack: 2024-09-11T14:40:52.278 INFO:teuthology.orchestra.run.smithi203.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x8b56c 0x10d76c 0xdd31a 0x4b14c 0x3b920e7 0x3b92268 0xb9f768b 0xb9fb0f9 0xba3eb1c 0xb9dfda2 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:52.278 INFO:teuthology.orchestra.run.smithi203.stderr:kernel callstack: 2024-09-11T14:40:52.335 INFO:teuthology.orchestra.run.smithi203.stderr:Reactor stalled for 123 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x10d767 0xdd31a 0x4b7ec 0xba391dc 0xb9e038f 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:52.336 INFO:teuthology.orchestra.run.smithi203.stderr:kernel callstack: 2024-09-11T14:40:52.376 INFO:teuthology.orchestra.run.smithi203.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x156dc 0xdd312 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:40:52.376 INFO:teuthology.orchestra.run.smithi203.stderr:kernel callstack: 2024-09-11T14:40:52.433 INFO:teuthology.orchestra.run.smithi203.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0xdd31f 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:40:52.433 INFO:teuthology.orchestra.run.smithi203.stderr:kernel callstack: 2024-09-11T14:40:52.469 INFO:teuthology.orchestra.run.smithi203.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x9e3b019 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:52.469 INFO:teuthology.orchestra.run.smithi203.stderr:kernel callstack: 2024-09-11T14:40:52.469 INFO:teuthology.orchestra.run.smithi203.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x5549f67 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:52.470 INFO:teuthology.orchestra.run.smithi203.stderr:kernel callstack: 2024-09-11T14:40:52.527 INFO:teuthology.orchestra.run.smithi203.stderr:Reactor stalled for 122 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x5549f89 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:52.527 INFO:teuthology.orchestra.run.smithi203.stderr:kernel callstack: 2024-09-11T14:40:52.836 INFO:teuthology.orchestra.run.smithi203.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x133c54 0xdd332 0x4ab1c 0xb9f07c3 0xb9f1bfa 0xb9f1e74 0xb9f201c 0xba39371 0xb9e038f 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93195d6 0x931a07a 0x931a0fd 0x931a182 0x931a284 0x931a37c 0x931a402 0x931a4bb 0x931a671 0x931a7a9 0x931a824 0x931a8a9 0x931aa04 0x931ab1e 0x931aba3 0x931ac81 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:52.836 INFO:teuthology.orchestra.run.smithi203.stderr:kernel callstack: 2024-09-11T14:40:52.994 INFO:teuthology.orchestra.run.smithi203.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x156e8 0xdd312 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:40:52.995 INFO:teuthology.orchestra.run.smithi203.stderr:kernel callstack: 2024-09-11T14:40:53.052 INFO:teuthology.orchestra.run.smithi203.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x10797d 0xdd305 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:40:53.052 INFO:teuthology.orchestra.run.smithi203.stderr:kernel callstack: 2024-09-11T14:40:53.913 INFO:teuthology.orchestra.run.smithi203.stderr:----------------------------------------------------- 2024-09-11T14:40:53.913 INFO:teuthology.orchestra.run.smithi203.stderr:Suppressions used: 2024-09-11T14:40:53.913 INFO:teuthology.orchestra.run.smithi203.stderr: count bytes template 2024-09-11T14:40:53.913 INFO:teuthology.orchestra.run.smithi203.stderr: 1 8 MallocExtension::Register 2024-09-11T14:40:53.913 INFO:teuthology.orchestra.run.smithi203.stderr:----------------------------------------------------- 2024-09-11T14:40:53.914 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-11T14:40:53.938 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-11T14:40:53.963 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:40:54.254 INFO:teuthology.orchestra.run.smithi203.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:40:54.255 INFO:teuthology.orchestra.run.smithi203.stderr:INFO 2024-09-11 14:40:54,254 seastar - Reactor backend: linux-aio 2024-09-11T14:40:54.260 INFO:teuthology.orchestra.run.smithi203.stderr:==27417==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:40:54.382 INFO:teuthology.orchestra.run.smithi203.stderr:INFO 2024-09-11 14:40:54,382 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-11T14:40:54.382 INFO:teuthology.orchestra.run.smithi203.stderr:INFO 2024-09-11 14:40:54,382 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 3-5 2024-09-11T14:40:54.383 INFO:teuthology.orchestra.run.smithi203.stderr:DEBUG 2024-09-11 14:40:54,383 [shard 0:main] seastar - reactor::drain 2024-09-11T14:40:54.886 INFO:teuthology.orchestra.run.smithi203.stderr:----------------------------------------------------- 2024-09-11T14:40:54.887 INFO:teuthology.orchestra.run.smithi203.stderr:Suppressions used: 2024-09-11T14:40:54.887 INFO:teuthology.orchestra.run.smithi203.stderr: count bytes template 2024-09-11T14:40:54.887 INFO:teuthology.orchestra.run.smithi203.stderr: 1 8 MallocExtension::Register 2024-09-11T14:40:54.887 INFO:teuthology.orchestra.run.smithi203.stderr:----------------------------------------------------- 2024-09-11T14:40:54.887 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-11T14:40:54.943 INFO:teuthology.orchestra.run.smithi203.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:40:54.947 INFO:teuthology.orchestra.run.smithi203.stderr:INFO 2024-09-11 14:40:54,946 seastar - Reactor backend: linux-aio 2024-09-11T14:40:54.956 INFO:teuthology.orchestra.run.smithi203.stderr:==27415==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:40:55.021 INFO:teuthology.orchestra.run.smithi203.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x361f77e 0x362d92e 0x5fc0c74 0x5e081e5 0x5e20427 0x715ebc2 0x7170395 0x7171832 0x71718b7 0x71719b9 0x7171ab1 0x7171cbe 0x7171ea4 0x71721f3 0x7172597 0x71727c7 0x7172a31 0x7172cfe 0x7172f1c 0x7172fd4 0x717309a 0x717317b 0x5c6b026 0x5c6b130 0x5c6b1ec 0x5c6b2a8 0x5c6b364 0x5c6b4b1 0x5c6b63c 0x5c6b9fd 0x5c85800 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:40:55.022 INFO:teuthology.orchestra.run.smithi203.stderr:kernel callstack: 2024-09-11T14:40:55.248 INFO:teuthology.orchestra.run.smithi203.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x4b1c5 0x3b920e7 0x3b92268 0xb9f0927 0xb9f1bfa 0xb9f1e74 0xb9f201c 0xb9f3dd7 0xb9f3fd3 0xba38ebd 0xb9e038f 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:55.248 INFO:teuthology.orchestra.run.smithi203.stderr:kernel callstack: 2024-09-11T14:40:55.249 INFO:teuthology.orchestra.run.smithi203.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x3b92189 0x3b92268 0xb9f0927 0xb9f1bfa 0xb9f1e74 0xb9f201c 0xb9f3dd7 0xb9f3fd3 0xba38ebd 0xb9e038f 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:55.249 INFO:teuthology.orchestra.run.smithi203.stderr:kernel callstack: 2024-09-11T14:40:55.306 INFO:teuthology.orchestra.run.smithi203.stderr:Reactor stalled for 90 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0xba3ec35 0xb9dfda2 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9ab9cb7 0x9b05946 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:55.306 INFO:teuthology.orchestra.run.smithi203.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x8b571 0x10d76c 0xdd31a 0x4ab1c 0x3acdafe 0x3b9225d 0xb9f768b 0xb9fb0f9 0xba3eb1c 0xb9e10b0 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b05f61 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:55.306 INFO:teuthology.orchestra.run.smithi203.stderr:kernel callstack: 2024-09-11T14:40:55.306 INFO:teuthology.orchestra.run.smithi203.stderr:kernel callstack: 2024-09-11T14:40:55.307 INFO:teuthology.orchestra.run.smithi203.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x40006 0x444c7 0xfb4fb 0x376ee9b 0xb9e0fe3 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:40:55.307 INFO:teuthology.orchestra.run.smithi203.stderr:kernel callstack: 2024-09-11T14:40:55.365 INFO:teuthology.orchestra.run.smithi203.stderr:Reactor stalled for 123 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x7d405 0x3883739 0xb9d9a87 0xb9dfb97 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:40:55.365 INFO:teuthology.orchestra.run.smithi203.stderr:kernel callstack: 2024-09-11T14:40:55.431 INFO:teuthology.orchestra.run.smithi203.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x5cca66b 0x5ccaaed 0x5ccaab2 0x5ccabe2 0xba24174 0xb9e408e 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:55.431 INFO:teuthology.orchestra.run.smithi203.stderr:kernel callstack: 2024-09-11T14:40:55.489 INFO:teuthology.orchestra.run.smithi203.stderr:Reactor stalled for 122 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x554a3be 0x5560890 0x9ed85bd 0x9ed8b76 0x9ed8f1e 0x9edc2f8 0x9edcf54 0x9edd217 0x9edde8e 0x9e293f2 0x9a07671 0x9a07c77 0x98fddfc 0x927c6a9 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:55.489 INFO:teuthology.orchestra.run.smithi203.stderr:kernel callstack: 2024-09-11T14:40:55.500 INFO:teuthology.orchestra.run.smithi203.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x554a11f 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:40:55.500 INFO:teuthology.orchestra.run.smithi203.stderr:kernel callstack: 2024-09-11T14:40:55.558 INFO:teuthology.orchestra.run.smithi203.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x5549a71 0x9e3b03b 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:40:55.558 INFO:teuthology.orchestra.run.smithi203.stderr:kernel callstack: 2024-09-11T14:40:55.608 INFO:teuthology.orchestra.run.smithi203.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x9e3aff4 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:55.608 INFO:teuthology.orchestra.run.smithi203.stderr:kernel callstack: 2024-09-11T14:40:55.608 INFO:teuthology.orchestra.run.smithi203.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0xdd2d8 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:55.608 INFO:teuthology.orchestra.run.smithi203.stderr:kernel callstack: 2024-09-11T14:40:55.665 INFO:teuthology.orchestra.run.smithi203.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x10798d 0xdd2fc 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:55.665 INFO:teuthology.orchestra.run.smithi203.stderr:kernel callstack: 2024-09-11T14:40:55.981 INFO:teuthology.orchestra.run.smithi203.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x10d767 0x10af7c 0x446e6 0xfb4fb 0x376ee9b 0x984af12 0x9b0161a 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93195d6 0x931a07a 0x931a0fd 0x931a182 0x931a284 0x931a37c 0x931a402 0x931a4bb 0x931a671 0x931a7a9 0x931a824 0x931a8a9 0x931aa04 0x931ab1e 0x931aba3 0x931ac81 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:40:55.981 INFO:teuthology.orchestra.run.smithi203.stderr:kernel callstack: 2024-09-11T14:40:57.042 INFO:teuthology.orchestra.run.smithi203.stderr:----------------------------------------------------- 2024-09-11T14:40:57.042 INFO:teuthology.orchestra.run.smithi203.stderr:Suppressions used: 2024-09-11T14:40:57.043 INFO:teuthology.orchestra.run.smithi203.stderr: count bytes template 2024-09-11T14:40:57.043 INFO:teuthology.orchestra.run.smithi203.stderr: 1 8 MallocExtension::Register 2024-09-11T14:40:57.043 INFO:teuthology.orchestra.run.smithi203.stderr:----------------------------------------------------- 2024-09-11T14:40:57.043 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-11T14:40:57.068 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-11T14:40:57.092 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-11T14:40:57.093 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-11T14:40:57.093 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-11T14:40:57.123 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-11T14:40:57.124 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-11T14:40:57.192 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-11T14:40:57.192 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-11T14:40:57.217 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-11T14:40:57.218 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-11T14:40:57.283 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-11T14:40:57.396 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-11T14:40:57.421 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-11T14:40:57.421 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-11T14:40:57.440 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-11T14:40:57.440 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-09-11T14:40:57.476 INFO:tasks.ceph:Adding keys to all mons... 2024-09-11T14:40:57.477 DEBUG:teuthology.orchestra.run.smithi181:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-11T14:40:57.509 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.0] 2024-09-11T14:40:57.509 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQDsq+FmRAbxBRAArEmiN2m03t/ssw2BEed9Tw== 2024-09-11T14:40:57.509 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.1] 2024-09-11T14:40:57.509 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQDvq+FmMf3hDRAAyDU1kRfJtJEbHsfe3ix0nQ== 2024-09-11T14:40:57.509 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr.x] 2024-09-11T14:40:57.509 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQDpq+Fm7I0mEBAAMUDiIMb4RvNJekFc2afrTw== 2024-09-11T14:40:57.509 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.2] 2024-09-11T14:40:57.509 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQDzq+FmWk1TOhAA8lGD8od48ridxhj/7fnmMQ== 2024-09-11T14:40:57.509 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.3] 2024-09-11T14:40:57.509 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQD3q+FmYkhLBhAAJ95NWyzXhi3SWWkzS3noWw== 2024-09-11T14:40:57.509 INFO:teuthology.orchestra.run.smithi181.stdout:[client.0] 2024-09-11T14:40:57.509 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQDpq+FmjStNExAALNLuXmZUaBduZQ1iUi44/Q== 2024-09-11T14:40:57.510 INFO:teuthology.orchestra.run.smithi181.stdout:[client.1] 2024-09-11T14:40:57.510 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQDpq+FmWFYWFxAAUDLeS1HoGdj9R9WK4FD+gA== 2024-09-11T14:40:57.510 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T14:40:57.621 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T14:40:57.684 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-11T14:40:57.746 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T14:40:57.807 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T14:40:57.867 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T14:40:57.932 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T14:40:57.996 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-11T14:40:57.997 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-11T14:40:58.021 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T14:40:58.134 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:40:58.133+0000 7f2bea835c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:40:58.134 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:40:58.133+0000 7f2bea835c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:40:58.148 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-11T14:40:58.172 DEBUG:teuthology.orchestra.run.smithi181:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:40:58.227 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-11T14:40:58.228 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-11T14:40:58.228 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-11T14:40:58.269 INFO:tasks.ceph.mon.a:Started 2024-09-11T14:40:58.270 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-11T14:40:58.270 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-11T14:40:58.270 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-11T14:40:58.272 INFO:tasks.ceph.mgr.x:Started 2024-09-11T14:40:58.272 DEBUG:tasks.ceph:set 0 configs 2024-09-11T14:40:58.273 INFO:tasks.ceph:Setting crush tunables to default 2024-09-11T14:40:58.273 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd crush tunables default 2024-09-11T14:40:58.349 INFO:tasks.ceph.mon.a.smithi181.stderr:2024-09-11T14:40:58.348+0000 7f65174efc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:40:58.350 INFO:tasks.ceph.mon.a.smithi181.stderr:2024-09-11T14:40:58.348+0000 7f65174efc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:40:58.350 INFO:tasks.ceph.mon.a.smithi181.stderr:2024-09-11T14:40:58.348+0000 7f65174efc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:40:58.373 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-09-11T14:40:58.371+0000 7f7561bd1000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:40:58.381 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:40:58.374+0000 7f72d66fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:40:58.386 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:40:58.384+0000 7f72d66fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:40:58.386 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-09-11T14:40:58.385+0000 7f7561bd1000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:40:58.386 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-09-11T14:40:58.385+0000 7f7561bd1000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:40:58.455 INFO:teuthology.orchestra.run.smithi181.stderr:adjusted tunables profile to default 2024-09-11T14:40:58.467 INFO:tasks.ceph:check_enable_crimson: True 2024-09-11T14:40:58.467 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-09-11T14:40:58.467 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-09-11T14:40:58.572 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:40:58.566+0000 7fab35d51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:40:58.576 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:40:58.575+0000 7fab35d51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:40:58.654 INFO:tasks.ceph.mon.a.smithi181.stderr:2024-09-11T14:40:58.652+0000 7f650e1f4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-09-11T14:40:58.654 INFO:tasks.ceph.mon.a.smithi181.stderr:Please be aware that this feature is experimental, untested, 2024-09-11T14:40:58.654 INFO:tasks.ceph.mon.a.smithi181.stderr:unsupported, and may result in data corruption, data loss, 2024-09-11T14:40:58.654 INFO:tasks.ceph.mon.a.smithi181.stderr:and/or irreparable damage to your cluster. Do not use 2024-09-11T14:40:58.654 INFO:tasks.ceph.mon.a.smithi181.stderr:feature with important data. 2024-09-11T14:40:58.654 INFO:tasks.ceph.mon.a.smithi181.stderr: 2024-09-11T14:40:59.470 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-11T14:40:59.471 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-11T14:40:59.471 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-11T14:40:59.506 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-11T14:40:59.507 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-11T14:40:59.573 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-11T14:40:59.573 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-11T14:40:59.598 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-11T14:40:59.598 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-11T14:40:59.663 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 05592c04-44ae-46d8-bcf7-a865d8397d5d 0 2024-09-11T14:40:59.811 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-11T14:40:59.804+0000 7fd0d5191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:40:59.814 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-11T14:40:59.813+0000 7fd0d5191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:40:59.892 INFO:teuthology.orchestra.run.smithi203.stdout:0 2024-09-11T14:40:59.904 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 70bdeffe-ad6e-4417-92f7-6021f541c690 1 2024-09-11T14:41:00.012 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-11T14:41:00.005+0000 7f307d6c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:00.015 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-11T14:41:00.014+0000 7f307d6c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:00.093 INFO:teuthology.orchestra.run.smithi203.stdout:1 2024-09-11T14:41:00.104 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new d3fd0624-0127-4f26-ba89-ca34a2c562ed 2 2024-09-11T14:41:00.213 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-11T14:41:00.205+0000 7f4a94b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:00.216 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-11T14:41:00.215+0000 7f4a94b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:00.297 INFO:teuthology.orchestra.run.smithi203.stdout:2 2024-09-11T14:41:00.308 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 24bbbdd2-a931-4cc2-a24d-61d131ece2c8 3 2024-09-11T14:41:00.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-11T14:41:00.409+0000 7f3cc4351640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:00.420 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-11T14:41:00.419+0000 7f3cc4351640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:00.498 INFO:teuthology.orchestra.run.smithi203.stdout:3 2024-09-11T14:41:00.511 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-09-11T14:41:00.541 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-11T14:41:00.541 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-09-11T14:41:00.583 INFO:tasks.ceph.osd.0:Started 2024-09-11T14:41:00.583 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-09-11T14:41:00.613 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-11T14:41:00.614 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-09-11T14:41:00.656 INFO:tasks.ceph.osd.1:Started 2024-09-11T14:41:00.656 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-09-11T14:41:00.686 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-11T14:41:00.687 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-09-11T14:41:00.729 INFO:tasks.ceph.osd.2:Started 2024-09-11T14:41:00.729 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.3.log 2024-09-11T14:41:00.758 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-11T14:41:00.758 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-09-11T14:41:00.800 INFO:tasks.ceph.osd.3:Started 2024-09-11T14:41:00.801 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T14:41:00.916 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:00.910+0000 7f50c7591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:00.919 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:00.918+0000 7f50c7591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:00.989 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:41:00.989 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":7,"fsid":"0f7e90de-623e-43a2-a836-b493e06f03fc","created":"2024-09-11T14:40:58.371084+0000","modified":"2024-09-11T14:41:00.494637+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T14:41:00.494637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"05592c04-44ae-46d8-bcf7-a865d8397d5d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"70bdeffe-ad6e-4417-92f7-6021f541c690","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3fd0624-0127-4f26-ba89-ca34a2c562ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"24bbbdd2-a931-4cc2-a24d-61d131ece2c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T14:41:01.000 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-11T14:41:01.001 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-11T14:41:01.403 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T14:41:01.508 INFO:teuthology.misc.health.smithi181.stderr:2024-09-11T14:41:01.503+0000 7fc39f991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:01.513 INFO:teuthology.misc.health.smithi181.stderr:2024-09-11T14:41:01.512+0000 7fc39f991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:01.581 INFO:teuthology.misc.health.smithi181.stdout: 2024-09-11T14:41:01.581 INFO:teuthology.misc.health.smithi181.stdout:{"epoch":7,"fsid":"0f7e90de-623e-43a2-a836-b493e06f03fc","created":"2024-09-11T14:40:58.371084+0000","modified":"2024-09-11T14:41:00.494637+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T14:41:00.494637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"05592c04-44ae-46d8-bcf7-a865d8397d5d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"70bdeffe-ad6e-4417-92f7-6021f541c690","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3fd0624-0127-4f26-ba89-ca34a2c562ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"24bbbdd2-a931-4cc2-a24d-61d131ece2c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T14:41:01.591 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-09-11T14:41:03.714 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-09-11T14:41:03.712+0000 7f74feef6640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-11T14:41:07.996 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T14:41:08.099 INFO:teuthology.misc.health.smithi181.stderr:2024-09-11T14:41:08.093+0000 7f647ad11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:08.103 INFO:teuthology.misc.health.smithi181.stderr:2024-09-11T14:41:08.102+0000 7f647ad11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:08.298 INFO:teuthology.misc.health.smithi181.stdout: 2024-09-11T14:41:08.298 INFO:teuthology.misc.health.smithi181.stdout:{"epoch":12,"fsid":"0f7e90de-623e-43a2-a836-b493e06f03fc","created":"2024-09-11T14:40:58.371084+0000","modified":"2024-09-11T14:41:07.775632+0000","last_up_change":"2024-09-11T14:41:05.766940+0000","last_in_change":"2024-09-11T14:41:00.494637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T14:41:06.733924+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"05592c04-44ae-46d8-bcf7-a865d8397d5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":920133847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6803","nonce":920133847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6809","nonce":920133847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6807","nonce":920133847}]},"public_addr":"172.21.15.181:6804/920133847","cluster_addr":"172.21.15.181:6803/920133847","heartbeat_back_addr":"172.21.15.181:6809/920133847","heartbeat_front_addr":"172.21.15.181:6807/920133847","state":["exists","up"]},{"osd":1,"uuid":"70bdeffe-ad6e-4417-92f7-6021f541c690","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6801","nonce":3969919471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3969919471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3969919471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6819","nonce":3969919471}]},"public_addr":"172.21.15.181:6801/3969919471","cluster_addr":"172.21.15.181:6802/3969919471","heartbeat_back_addr":"172.21.15.181:6820/3969919471","heartbeat_front_addr":"172.21.15.181:6819/3969919471","state":["exists","up"]},{"osd":2,"uuid":"d3fd0624-0127-4f26-ba89-ca34a2c562ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6811","nonce":692234911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":692234911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":692234911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":692234911}]},"public_addr":"172.21.15.203:6811/692234911","cluster_addr":"172.21.15.203:6810/692234911","heartbeat_back_addr":"172.21.15.203:6808/692234911","heartbeat_front_addr":"172.21.15.203:6809/692234911","state":["exists","up"]},{"osd":3,"uuid":"24bbbdd2-a931-4cc2-a24d-61d131ece2c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6803","nonce":667507199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":667507199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":667507199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6805","nonce":667507199}]},"public_addr":"172.21.15.203:6803/667507199","cluster_addr":"172.21.15.203:6802/667507199","heartbeat_back_addr":"172.21.15.203:6804/667507199","heartbeat_front_addr":"172.21.15.203:6805/667507199","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T14:41:08.310 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-09-11T14:41:08.311 INFO:tasks.ceph:Creating RBD pool 2024-09-11T14:41:08.311 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-11T14:41:08.422 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:08.413+0000 7ff31dd91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:08.429 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:08.428+0000 7ff31dd91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:08.787 INFO:teuthology.orchestra.run.smithi181.stderr:pool 'rbd' created 2024-09-11T14:41:08.808 DEBUG:teuthology.orchestra.run.smithi181:> rbd --cluster ceph pool init rbd 2024-09-11T14:41:08.911 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:08.910+0000 7f7322602ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:08.911 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:08.910+0000 7f7322602ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:08.916 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:08.915+0000 7f7322602ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:11.857 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-11T14:41:11.858 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-11T14:41:11.859 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-11T14:41:11.968 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:11.962+0000 7fea4664f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:11.972 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:11.971+0000 7fea4664f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:12.158 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:41:12.171 INFO:teuthology.orchestra.run.smithi181.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2024-09-11T14:41:12.172 INFO:tasks.ceph_manager:config epoch is 1 2024-09-11T14:41:12.172 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-11T14:41:12.172 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-11T14:41:12.173 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-11T14:41:12.281 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:12.276+0000 7facef991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:12.285 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:12.284+0000 7facef991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:12.516 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:41:12.528 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":1940838471},{"type":"v1","addr":"172.21.15.203:6801","nonce":1940838471}]},"active_addr":"172.21.15.203:6801/1940838471","active_change":"2024-09-11T14:41:02.697097+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.203:0","nonce":2801457454}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.203:0","nonce":4019446592}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.203:0","nonce":1590901936}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.203:0","nonce":1936234314}]}]} 2024-09-11T14:41:12.532 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-11T14:41:12.532 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-11T14:41:12.532 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T14:41:12.641 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:12.635+0000 7fcf658c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:12.644 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:12.643+0000 7fcf658c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:12.830 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:41:12.830 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":16,"fsid":"0f7e90de-623e-43a2-a836-b493e06f03fc","created":"2024-09-11T14:40:58.371084+0000","modified":"2024-09-11T14:41:11.795653+0000","last_up_change":"2024-09-11T14:41:05.766940+0000","last_in_change":"2024-09-11T14:41:00.494637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T14:41:06.733924+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T14:41:08.627066+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"05592c04-44ae-46d8-bcf7-a865d8397d5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":920133847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6803","nonce":920133847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6809","nonce":920133847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6807","nonce":920133847}]},"public_addr":"172.21.15.181:6804/920133847","cluster_addr":"172.21.15.181:6803/920133847","heartbeat_back_addr":"172.21.15.181:6809/920133847","heartbeat_front_addr":"172.21.15.181:6807/920133847","state":["exists","up"]},{"osd":1,"uuid":"70bdeffe-ad6e-4417-92f7-6021f541c690","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6801","nonce":3969919471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3969919471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3969919471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6819","nonce":3969919471}]},"public_addr":"172.21.15.181:6801/3969919471","cluster_addr":"172.21.15.181:6802/3969919471","heartbeat_back_addr":"172.21.15.181:6820/3969919471","heartbeat_front_addr":"172.21.15.181:6819/3969919471","state":["exists","up"]},{"osd":2,"uuid":"d3fd0624-0127-4f26-ba89-ca34a2c562ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6811","nonce":692234911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":692234911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":692234911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":692234911}]},"public_addr":"172.21.15.203:6811/692234911","cluster_addr":"172.21.15.203:6810/692234911","heartbeat_back_addr":"172.21.15.203:6808/692234911","heartbeat_front_addr":"172.21.15.203:6809/692234911","state":["exists","up"]},{"osd":3,"uuid":"24bbbdd2-a931-4cc2-a24d-61d131ece2c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6803","nonce":667507199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":667507199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":667507199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6805","nonce":667507199}]},"public_addr":"172.21.15.203:6803/667507199","cluster_addr":"172.21.15.203:6802/667507199","heartbeat_back_addr":"172.21.15.203:6804/667507199","heartbeat_front_addr":"172.21.15.203:6805/667507199","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T14:41:12.845 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-11T14:41:12.845 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T14:41:12.953 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:12.947+0000 7f3cd12fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:12.956 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:12.955+0000 7f3cd12fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:13.140 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:41:13.141 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":16,"fsid":"0f7e90de-623e-43a2-a836-b493e06f03fc","created":"2024-09-11T14:40:58.371084+0000","modified":"2024-09-11T14:41:11.795653+0000","last_up_change":"2024-09-11T14:41:05.766940+0000","last_in_change":"2024-09-11T14:41:00.494637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T14:41:06.733924+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T14:41:08.627066+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"05592c04-44ae-46d8-bcf7-a865d8397d5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":920133847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6803","nonce":920133847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6809","nonce":920133847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6807","nonce":920133847}]},"public_addr":"172.21.15.181:6804/920133847","cluster_addr":"172.21.15.181:6803/920133847","heartbeat_back_addr":"172.21.15.181:6809/920133847","heartbeat_front_addr":"172.21.15.181:6807/920133847","state":["exists","up"]},{"osd":1,"uuid":"70bdeffe-ad6e-4417-92f7-6021f541c690","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6801","nonce":3969919471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3969919471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3969919471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6819","nonce":3969919471}]},"public_addr":"172.21.15.181:6801/3969919471","cluster_addr":"172.21.15.181:6802/3969919471","heartbeat_back_addr":"172.21.15.181:6820/3969919471","heartbeat_front_addr":"172.21.15.181:6819/3969919471","state":["exists","up"]},{"osd":2,"uuid":"d3fd0624-0127-4f26-ba89-ca34a2c562ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6811","nonce":692234911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":692234911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":692234911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":692234911}]},"public_addr":"172.21.15.203:6811/692234911","cluster_addr":"172.21.15.203:6810/692234911","heartbeat_back_addr":"172.21.15.203:6808/692234911","heartbeat_front_addr":"172.21.15.203:6809/692234911","state":["exists","up"]},{"osd":3,"uuid":"24bbbdd2-a931-4cc2-a24d-61d131ece2c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6803","nonce":667507199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":667507199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":667507199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6805","nonce":667507199}]},"public_addr":"172.21.15.203:6803/667507199","cluster_addr":"172.21.15.203:6802/667507199","heartbeat_back_addr":"172.21.15.203:6804/667507199","heartbeat_front_addr":"172.21.15.203:6805/667507199","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T14:41:13.155 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-11T14:41:13.155 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-11T14:41:13.155 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-11T14:41:13.156 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-11T14:41:13.277 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:13.271+0000 7f042226c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:13.278 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:13.271+0000 7f865ae97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:13.278 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:13.271+0000 7f88c4d51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:13.278 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:13.271+0000 7f7244d11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:13.282 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:13.281+0000 7f7244d11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:13.282 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:13.281+0000 7f865ae97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:13.282 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:13.281+0000 7f042226c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:13.283 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:13.282+0000 7f88c4d51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:13.317 INFO:teuthology.orchestra.run.smithi181.stdout:38654705672 2024-09-11T14:41:13.317 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T14:41:13.318 INFO:teuthology.orchestra.run.smithi181.stdout:42949672967 2024-09-11T14:41:13.319 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T14:41:13.322 INFO:teuthology.orchestra.run.smithi181.stdout:38654705672 2024-09-11T14:41:13.323 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T14:41:13.323 INFO:teuthology.orchestra.run.smithi181.stdout:38654705672 2024-09-11T14:41:13.324 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T14:41:13.431 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:13.425+0000 7f115e551640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:13.437 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:13.435+0000 7f115e551640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:13.484 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:13.475+0000 7fd0456d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:13.485 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:13.478+0000 7ff1df05a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:13.488 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:13.487+0000 7ff1df05a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:13.489 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:13.488+0000 7fd0456d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:13.498 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:13.490+0000 7ff91f111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:13.506 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:13.505+0000 7ff91f111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:13.685 INFO:teuthology.orchestra.run.smithi181.stdout:38654705668 2024-09-11T14:41:13.699 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705672 got 38654705668 for osd.0 2024-09-11T14:41:13.728 INFO:teuthology.orchestra.run.smithi181.stdout:38654705668 2024-09-11T14:41:13.731 INFO:teuthology.orchestra.run.smithi181.stdout:42949672964 2024-09-11T14:41:13.734 INFO:teuthology.orchestra.run.smithi181.stdout:38654705668 2024-09-11T14:41:13.743 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705672 got 38654705668 for osd.3 2024-09-11T14:41:13.747 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672967 got 42949672964 for osd.2 2024-09-11T14:41:13.749 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705672 got 38654705668 for osd.1 2024-09-11T14:41:14.701 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T14:41:14.744 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T14:41:14.747 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T14:41:14.750 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T14:41:14.830 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:14.822+0000 7f7206911640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:14.837 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:14.835+0000 7f72056af640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:14.873 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:14.863+0000 7f8a9e391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:14.877 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:14.876+0000 7f8a9e391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:14.908 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:14.902+0000 7f7244d11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:14.913 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:14.906+0000 7f1813d51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:14.913 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:14.911+0000 7f7244d11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:14.917 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:14.916+0000 7f1813d51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:15.047 INFO:teuthology.orchestra.run.smithi181.stdout:38654705672 2024-09-11T14:41:15.060 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705672 got 38654705672 for osd.0 2024-09-11T14:41:15.061 DEBUG:teuthology.parallel:result is None 2024-09-11T14:41:15.083 INFO:teuthology.orchestra.run.smithi181.stdout:38654705672 2024-09-11T14:41:15.097 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705672 got 38654705672 for osd.3 2024-09-11T14:41:15.097 DEBUG:teuthology.parallel:result is None 2024-09-11T14:41:15.113 INFO:teuthology.orchestra.run.smithi181.stdout:42949672967 2024-09-11T14:41:15.127 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672967 got 42949672967 for osd.2 2024-09-11T14:41:15.127 DEBUG:teuthology.parallel:result is None 2024-09-11T14:41:15.130 INFO:teuthology.orchestra.run.smithi181.stdout:38654705672 2024-09-11T14:41:15.143 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705672 got 38654705672 for osd.1 2024-09-11T14:41:15.143 DEBUG:teuthology.parallel:result is None 2024-09-11T14:41:15.144 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-11T14:41:15.144 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:41:15.293 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:15.287+0000 7f6f7b151640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:15.296 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:15.295+0000 7f6f7b151640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:15.480 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:41:15.480 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:41:15.493 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-09-11T14:41:14.706245+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":5112,"kb_used_data":5112,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209185800,"statfs":{"total":214211493888,"available":214206259200,"internally_reserved":0,"allocated":5234688,"data_stored":5234688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.910173"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.828733+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:41:11.828733+0000","last_peered":"2024-09-11T14:41:11.828733+0000","last_clean":"2024-09-11T14:41:11.828733+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:41:11.828733+0000","last_undegraded":"2024-09-11T14:41:11.828733+0000","last_fullsized":"2024-09-11T14:41:11.828733+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.828226+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:41:11.828226+0000","last_peered":"2024-09-11T14:41:11.828226+0000","last_clean":"2024-09-11T14:41:11.828226+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:41:11.828226+0000","last_undegraded":"2024-09-11T14:41:11.828226+0000","last_fullsized":"2024-09-11T14:41:11.828226+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.826327+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:41:11.826327+0000","last_peered":"2024-09-11T14:41:11.826327+0000","last_clean":"2024-09-11T14:41:11.826327+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:41:11.826327+0000","last_undegraded":"2024-09-11T14:41:11.826327+0000","last_fullsized":"2024-09-11T14:41:11.826327+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.828914+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:41:11.828914+0000","last_peered":"2024-09-11T14:41:11.828914+0000","last_clean":"2024-09-11T14:41:11.828914+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:41:11.828914+0000","last_undegraded":"2024-09-11T14:41:11.828914+0000","last_fullsized":"2024-09-11T14:41:11.828914+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.827914+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:41:11.827914+0000","last_peered":"2024-09-11T14:41:11.827914+0000","last_clean":"2024-09-11T14:41:11.827914+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:41:11.827914+0000","last_undegraded":"2024-09-11T14:41:11.827914+0000","last_fullsized":"2024-09-11T14:41:11.827914+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.822076+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:41:11.822076+0000","last_peered":"2024-09-11T14:41:11.822076+0000","last_clean":"2024-09-11T14:41:11.822076+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:41:11.822076+0000","last_undegraded":"2024-09-11T14:41:11.822076+0000","last_fullsized":"2024-09-11T14:41:11.822076+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.824558+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:41:11.824558+0000","last_peered":"2024-09-11T14:41:11.824558+0000","last_clean":"2024-09-11T14:41:11.824558+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:41:11.824558+0000","last_undegraded":"2024-09-11T14:41:11.824558+0000","last_fullsized":"2024-09-11T14:41:11.824558+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.828717+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:41:11.828717+0000","last_peered":"2024-09-11T14:41:11.828717+0000","last_clean":"2024-09-11T14:41:11.828717+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:41:11.828717+0000","last_undegraded":"2024-09-11T14:41:11.828717+0000","last_fullsized":"2024-09-11T14:41:11.828717+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":88,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.824687+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:41:11.824687+0000","last_peered":"2024-09-11T14:41:11.824687+0000","last_clean":"2024-09-11T14:41:11.824687+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:41:11.824687+0000","last_undegraded":"2024-09-11T14:41:11.824687+0000","last_fullsized":"2024-09-11T14:41:11.824687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214206259200,"internally_reserved":0,"allocated":5234688,"data_stored":5234688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551599616,"internally_reserved":0,"allocated":1273856,"data_stored":1273856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672967,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1632,"kb_used_data":1632,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296096,"statfs":{"total":53552873472,"available":53551202304,"internally_reserved":0,"allocated":1671168,"data_stored":1671168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1244,"kb_used_data":1244,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296484,"statfs":{"total":53552873472,"available":53551599616,"internally_reserved":0,"allocated":1273856,"data_stored":1273856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":668,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297060,"statfs":{"total":53552873472,"available":53552189440,"internally_reserved":0,"allocated":684032,"data_stored":684032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1568,"kb_used_data":1568,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296160,"statfs":{"total":53552873472,"available":53551267840,"internally_reserved":0,"allocated":1605632,"data_stored":1605632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53551599616,"internally_reserved":0,"allocated":1273856,"data_stored":1273856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551267840,"internally_reserved":0,"allocated":1605632,"data_stored":1605632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53552189440,"internally_reserved":0,"allocated":684032,"data_stored":684032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53551202304,"internally_reserved":0,"allocated":1671168,"data_stored":1671168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53551599616,"internally_reserved":0,"allocated":1273856,"data_stored":1273856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:41:15.494 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:41:15.602 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:15.597+0000 7f7e1a391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:15.606 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:15.604+0000 7f7e1a391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:15.788 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:41:15.788 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:41:15.800 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-09-11T14:41:14.706245+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":5112,"kb_used_data":5112,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209185800,"statfs":{"total":214211493888,"available":214206259200,"internally_reserved":0,"allocated":5234688,"data_stored":5234688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.910173"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.828733+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:41:11.828733+0000","last_peered":"2024-09-11T14:41:11.828733+0000","last_clean":"2024-09-11T14:41:11.828733+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:41:11.828733+0000","last_undegraded":"2024-09-11T14:41:11.828733+0000","last_fullsized":"2024-09-11T14:41:11.828733+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.828226+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:41:11.828226+0000","last_peered":"2024-09-11T14:41:11.828226+0000","last_clean":"2024-09-11T14:41:11.828226+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:41:11.828226+0000","last_undegraded":"2024-09-11T14:41:11.828226+0000","last_fullsized":"2024-09-11T14:41:11.828226+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.826327+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:41:11.826327+0000","last_peered":"2024-09-11T14:41:11.826327+0000","last_clean":"2024-09-11T14:41:11.826327+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:41:11.826327+0000","last_undegraded":"2024-09-11T14:41:11.826327+0000","last_fullsized":"2024-09-11T14:41:11.826327+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.828914+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:41:11.828914+0000","last_peered":"2024-09-11T14:41:11.828914+0000","last_clean":"2024-09-11T14:41:11.828914+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:41:11.828914+0000","last_undegraded":"2024-09-11T14:41:11.828914+0000","last_fullsized":"2024-09-11T14:41:11.828914+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.827914+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:41:11.827914+0000","last_peered":"2024-09-11T14:41:11.827914+0000","last_clean":"2024-09-11T14:41:11.827914+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:41:11.827914+0000","last_undegraded":"2024-09-11T14:41:11.827914+0000","last_fullsized":"2024-09-11T14:41:11.827914+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.822076+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:41:11.822076+0000","last_peered":"2024-09-11T14:41:11.822076+0000","last_clean":"2024-09-11T14:41:11.822076+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:41:11.822076+0000","last_undegraded":"2024-09-11T14:41:11.822076+0000","last_fullsized":"2024-09-11T14:41:11.822076+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.824558+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:41:11.824558+0000","last_peered":"2024-09-11T14:41:11.824558+0000","last_clean":"2024-09-11T14:41:11.824558+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:41:11.824558+0000","last_undegraded":"2024-09-11T14:41:11.824558+0000","last_fullsized":"2024-09-11T14:41:11.824558+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.828717+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:41:11.828717+0000","last_peered":"2024-09-11T14:41:11.828717+0000","last_clean":"2024-09-11T14:41:11.828717+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:41:11.828717+0000","last_undegraded":"2024-09-11T14:41:11.828717+0000","last_fullsized":"2024-09-11T14:41:11.828717+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":88,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.824687+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:41:11.824687+0000","last_peered":"2024-09-11T14:41:11.824687+0000","last_clean":"2024-09-11T14:41:11.824687+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:41:11.824687+0000","last_undegraded":"2024-09-11T14:41:11.824687+0000","last_fullsized":"2024-09-11T14:41:11.824687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214206259200,"internally_reserved":0,"allocated":5234688,"data_stored":5234688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551599616,"internally_reserved":0,"allocated":1273856,"data_stored":1273856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672967,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1632,"kb_used_data":1632,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296096,"statfs":{"total":53552873472,"available":53551202304,"internally_reserved":0,"allocated":1671168,"data_stored":1671168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1244,"kb_used_data":1244,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296484,"statfs":{"total":53552873472,"available":53551599616,"internally_reserved":0,"allocated":1273856,"data_stored":1273856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":668,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297060,"statfs":{"total":53552873472,"available":53552189440,"internally_reserved":0,"allocated":684032,"data_stored":684032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1568,"kb_used_data":1568,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296160,"statfs":{"total":53552873472,"available":53551267840,"internally_reserved":0,"allocated":1605632,"data_stored":1605632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53551599616,"internally_reserved":0,"allocated":1273856,"data_stored":1273856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551267840,"internally_reserved":0,"allocated":1605632,"data_stored":1605632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53552189440,"internally_reserved":0,"allocated":684032,"data_stored":684032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53551202304,"internally_reserved":0,"allocated":1671168,"data_stored":1671168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53551599616,"internally_reserved":0,"allocated":1273856,"data_stored":1273856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:41:15.801 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-11T14:41:15.801 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-11T14:41:15.801 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-11T14:41:15.802 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-11T14:41:15.911 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:15.907+0000 7f4e3727d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:15.915 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:15.913+0000 7f4e3727d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:16.135 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:41:16.135 INFO:teuthology.orchestra.run.smithi181.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-11T14:41:16.148 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-11T14:41:16.149 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-11T14:41:16.159 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_down': 0, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-11T14:41:16.160 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client.0': {'admin socket': '/var/run/ceph/ceph-$name.asok'}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a'}}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-matanb-crimson-seastore-logs', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'dbebf1a3db6db7b5579464ee715edcf1b3e9cce2'}} 2024-09-11T14:41:16.160 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_down': 0, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 4, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-11T14:41:16.160 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-11T14:41:16.271 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:16.262+0000 7f711c991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:16.274 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:16.273+0000 7f711c991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:16.479 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-11T14:41:16.480 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-11T14:41:16.480 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:41:16.587 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:16.582+0000 7fcb53591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:16.591 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:16.589+0000 7fcb53591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:16.775 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:41:16.775 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:41:16.788 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-09-11T14:41:16.706670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":5112,"kb_used_data":5112,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209185800,"statfs":{"total":214211493888,"available":214206259200,"internally_reserved":0,"allocated":5234688,"data_stored":5234688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.910597"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.828733+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:41:11.828733+0000","last_peered":"2024-09-11T14:41:11.828733+0000","last_clean":"2024-09-11T14:41:11.828733+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:41:11.828733+0000","last_undegraded":"2024-09-11T14:41:11.828733+0000","last_fullsized":"2024-09-11T14:41:11.828733+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.828226+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:41:11.828226+0000","last_peered":"2024-09-11T14:41:11.828226+0000","last_clean":"2024-09-11T14:41:11.828226+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:41:11.828226+0000","last_undegraded":"2024-09-11T14:41:11.828226+0000","last_fullsized":"2024-09-11T14:41:11.828226+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.826327+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:41:11.826327+0000","last_peered":"2024-09-11T14:41:11.826327+0000","last_clean":"2024-09-11T14:41:11.826327+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:41:11.826327+0000","last_undegraded":"2024-09-11T14:41:11.826327+0000","last_fullsized":"2024-09-11T14:41:11.826327+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.828914+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:41:11.828914+0000","last_peered":"2024-09-11T14:41:11.828914+0000","last_clean":"2024-09-11T14:41:11.828914+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:41:11.828914+0000","last_undegraded":"2024-09-11T14:41:11.828914+0000","last_fullsized":"2024-09-11T14:41:11.828914+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.827914+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:41:11.827914+0000","last_peered":"2024-09-11T14:41:11.827914+0000","last_clean":"2024-09-11T14:41:11.827914+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:41:11.827914+0000","last_undegraded":"2024-09-11T14:41:11.827914+0000","last_fullsized":"2024-09-11T14:41:11.827914+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.822076+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:41:11.822076+0000","last_peered":"2024-09-11T14:41:11.822076+0000","last_clean":"2024-09-11T14:41:11.822076+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:41:11.822076+0000","last_undegraded":"2024-09-11T14:41:11.822076+0000","last_fullsized":"2024-09-11T14:41:11.822076+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.824558+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:41:11.824558+0000","last_peered":"2024-09-11T14:41:11.824558+0000","last_clean":"2024-09-11T14:41:11.824558+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:41:11.824558+0000","last_undegraded":"2024-09-11T14:41:11.824558+0000","last_fullsized":"2024-09-11T14:41:11.824558+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.828717+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:41:11.828717+0000","last_peered":"2024-09-11T14:41:11.828717+0000","last_clean":"2024-09-11T14:41:11.828717+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:41:11.828717+0000","last_undegraded":"2024-09-11T14:41:11.828717+0000","last_fullsized":"2024-09-11T14:41:11.828717+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":88,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.824687+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:41:11.824687+0000","last_peered":"2024-09-11T14:41:11.824687+0000","last_clean":"2024-09-11T14:41:11.824687+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:41:11.824687+0000","last_undegraded":"2024-09-11T14:41:11.824687+0000","last_fullsized":"2024-09-11T14:41:11.824687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214206259200,"internally_reserved":0,"allocated":5234688,"data_stored":5234688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551599616,"internally_reserved":0,"allocated":1273856,"data_stored":1273856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1632,"kb_used_data":1632,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296096,"statfs":{"total":53552873472,"available":53551202304,"internally_reserved":0,"allocated":1671168,"data_stored":1671168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1244,"kb_used_data":1244,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296484,"statfs":{"total":53552873472,"available":53551599616,"internally_reserved":0,"allocated":1273856,"data_stored":1273856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":668,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297060,"statfs":{"total":53552873472,"available":53552189440,"internally_reserved":0,"allocated":684032,"data_stored":684032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1568,"kb_used_data":1568,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296160,"statfs":{"total":53552873472,"available":53551267840,"internally_reserved":0,"allocated":1605632,"data_stored":1605632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53551599616,"internally_reserved":0,"allocated":1273856,"data_stored":1273856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551267840,"internally_reserved":0,"allocated":1605632,"data_stored":1605632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53552189440,"internally_reserved":0,"allocated":684032,"data_stored":684032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53551202304,"internally_reserved":0,"allocated":1671168,"data_stored":1671168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53551599616,"internally_reserved":0,"allocated":1273856,"data_stored":1273856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:41:16.789 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:41:16.898 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:16.893+0000 7f78fd074640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:16.902 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:16.900+0000 7f78fd074640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:17.085 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:41:17.086 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:41:17.099 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-09-11T14:41:16.706670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":5112,"kb_used_data":5112,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209185800,"statfs":{"total":214211493888,"available":214206259200,"internally_reserved":0,"allocated":5234688,"data_stored":5234688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.910597"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.828733+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:41:11.828733+0000","last_peered":"2024-09-11T14:41:11.828733+0000","last_clean":"2024-09-11T14:41:11.828733+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:41:11.828733+0000","last_undegraded":"2024-09-11T14:41:11.828733+0000","last_fullsized":"2024-09-11T14:41:11.828733+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.828226+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:41:11.828226+0000","last_peered":"2024-09-11T14:41:11.828226+0000","last_clean":"2024-09-11T14:41:11.828226+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:41:11.828226+0000","last_undegraded":"2024-09-11T14:41:11.828226+0000","last_fullsized":"2024-09-11T14:41:11.828226+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.826327+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:41:11.826327+0000","last_peered":"2024-09-11T14:41:11.826327+0000","last_clean":"2024-09-11T14:41:11.826327+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:41:11.826327+0000","last_undegraded":"2024-09-11T14:41:11.826327+0000","last_fullsized":"2024-09-11T14:41:11.826327+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.828914+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:41:11.828914+0000","last_peered":"2024-09-11T14:41:11.828914+0000","last_clean":"2024-09-11T14:41:11.828914+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:41:11.828914+0000","last_undegraded":"2024-09-11T14:41:11.828914+0000","last_fullsized":"2024-09-11T14:41:11.828914+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.827914+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:41:11.827914+0000","last_peered":"2024-09-11T14:41:11.827914+0000","last_clean":"2024-09-11T14:41:11.827914+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:41:11.827914+0000","last_undegraded":"2024-09-11T14:41:11.827914+0000","last_fullsized":"2024-09-11T14:41:11.827914+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.822076+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:41:11.822076+0000","last_peered":"2024-09-11T14:41:11.822076+0000","last_clean":"2024-09-11T14:41:11.822076+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:41:11.822076+0000","last_undegraded":"2024-09-11T14:41:11.822076+0000","last_fullsized":"2024-09-11T14:41:11.822076+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.824558+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:41:11.824558+0000","last_peered":"2024-09-11T14:41:11.824558+0000","last_clean":"2024-09-11T14:41:11.824558+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:41:11.824558+0000","last_undegraded":"2024-09-11T14:41:11.824558+0000","last_fullsized":"2024-09-11T14:41:11.824558+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.828717+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:41:11.828717+0000","last_peered":"2024-09-11T14:41:11.828717+0000","last_clean":"2024-09-11T14:41:11.828717+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:41:11.828717+0000","last_undegraded":"2024-09-11T14:41:11.828717+0000","last_fullsized":"2024-09-11T14:41:11.828717+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":88,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T14:41:11.824687+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:41:11.824687+0000","last_peered":"2024-09-11T14:41:11.824687+0000","last_clean":"2024-09-11T14:41:11.824687+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:41:11.824687+0000","last_undegraded":"2024-09-11T14:41:11.824687+0000","last_fullsized":"2024-09-11T14:41:11.824687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214206259200,"internally_reserved":0,"allocated":5234688,"data_stored":5234688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551599616,"internally_reserved":0,"allocated":1273856,"data_stored":1273856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1632,"kb_used_data":1632,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296096,"statfs":{"total":53552873472,"available":53551202304,"internally_reserved":0,"allocated":1671168,"data_stored":1671168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1244,"kb_used_data":1244,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296484,"statfs":{"total":53552873472,"available":53551599616,"internally_reserved":0,"allocated":1273856,"data_stored":1273856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":668,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297060,"statfs":{"total":53552873472,"available":53552189440,"internally_reserved":0,"allocated":684032,"data_stored":684032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1568,"kb_used_data":1568,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296160,"statfs":{"total":53552873472,"available":53551267840,"internally_reserved":0,"allocated":1605632,"data_stored":1605632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53551599616,"internally_reserved":0,"allocated":1273856,"data_stored":1273856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551267840,"internally_reserved":0,"allocated":1605632,"data_stored":1605632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53552189440,"internally_reserved":0,"allocated":684032,"data_stored":684032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53551202304,"internally_reserved":0,"allocated":1671168,"data_stored":1671168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53551599616,"internally_reserved":0,"allocated":1273856,"data_stored":1273856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:41:17.099 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-11T14:41:17.100 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-11T14:41:17.207 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:17.202+0000 7f2fe9abd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:17.211 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:17.210+0000 7f2fe9abd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:17.395 INFO:teuthology.orchestra.run.smithi181.stdout:epoch 16 2024-09-11T14:41:17.395 INFO:teuthology.orchestra.run.smithi181.stdout:fsid 0f7e90de-623e-43a2-a836-b493e06f03fc 2024-09-11T14:41:17.395 INFO:teuthology.orchestra.run.smithi181.stdout:created 2024-09-11T14:40:58.371084+0000 2024-09-11T14:41:17.395 INFO:teuthology.orchestra.run.smithi181.stdout:modified 2024-09-11T14:41:11.795653+0000 2024-09-11T14:41:17.395 INFO:teuthology.orchestra.run.smithi181.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-11T14:41:17.395 INFO:teuthology.orchestra.run.smithi181.stdout:crush_version 4 2024-09-11T14:41:17.395 INFO:teuthology.orchestra.run.smithi181.stdout:full_ratio 0.95 2024-09-11T14:41:17.395 INFO:teuthology.orchestra.run.smithi181.stdout:backfillfull_ratio 0.9 2024-09-11T14:41:17.395 INFO:teuthology.orchestra.run.smithi181.stdout:nearfull_ratio 0.85 2024-09-11T14:41:17.396 INFO:teuthology.orchestra.run.smithi181.stdout:require_min_compat_client luminous 2024-09-11T14:41:17.396 INFO:teuthology.orchestra.run.smithi181.stdout:min_compat_client jewel 2024-09-11T14:41:17.396 INFO:teuthology.orchestra.run.smithi181.stdout:require_osd_release squid 2024-09-11T14:41:17.396 INFO:teuthology.orchestra.run.smithi181.stdout:stretch_mode_enabled false 2024-09-11T14:41:17.396 INFO:teuthology.orchestra.run.smithi181.stdout:allow_crimson=true 2024-09-11T14:41:17.396 INFO:teuthology.orchestra.run.smithi181.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 13 flags hashpspool,nopgchange,crimson stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 4.00 2024-09-11T14:41:17.396 INFO:teuthology.orchestra.run.smithi181.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 16 flags hashpspool,nopgchange,selfmanaged_snaps,crimson stripe_width 0 application rbd read_balance_score 1.50 2024-09-11T14:41:17.396 INFO:teuthology.orchestra.run.smithi181.stdout: removed_snaps_queue [2~1] 2024-09-11T14:41:17.396 INFO:teuthology.orchestra.run.smithi181.stdout:max_osd 4 2024-09-11T14:41:17.396 INFO:teuthology.orchestra.run.smithi181.stdout:osd.0 up in weight 1 up_from 9 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.181:6804/920133847 v2:172.21.15.181:6803/920133847 exists,up 05592c04-44ae-46d8-bcf7-a865d8397d5d 2024-09-11T14:41:17.396 INFO:teuthology.orchestra.run.smithi181.stdout:osd.1 up in weight 1 up_from 9 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.181:6801/3969919471 v2:172.21.15.181:6802/3969919471 exists,up 70bdeffe-ad6e-4417-92f7-6021f541c690 2024-09-11T14:41:17.397 INFO:teuthology.orchestra.run.smithi181.stdout:osd.2 up in weight 1 up_from 10 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.203:6811/692234911 v2:172.21.15.203:6810/692234911 exists,up d3fd0624-0127-4f26-ba89-ca34a2c562ed 2024-09-11T14:41:17.397 INFO:teuthology.orchestra.run.smithi181.stdout:osd.3 up in weight 1 up_from 9 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.203:6803/667507199 v2:172.21.15.203:6802/667507199 exists,up 24bbbdd2-a931-4cc2-a24d-61d131ece2c8 2024-09-11T14:41:17.409 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 9 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.181:6804/920133847 v2:172.21.15.181:6803/920133847 exists,up 05592c04-44ae-46d8-bcf7-a865d8397d5d', 'osd.1 up in weight 1 up_from 9 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.181:6801/3969919471 v2:172.21.15.181:6802/3969919471 exists,up 70bdeffe-ad6e-4417-92f7-6021f541c690', 'osd.2 up in weight 1 up_from 10 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.203:6811/692234911 v2:172.21.15.203:6810/692234911 exists,up d3fd0624-0127-4f26-ba89-ca34a2c562ed', 'osd.3 up in weight 1 up_from 9 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.203:6803/667507199 v2:172.21.15.203:6802/667507199 exists,up 24bbbdd2-a931-4cc2-a24d-61d131ece2c8'] 2024-09-11T14:41:17.409 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-09-11T14:41:17.514 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T14:41:17.514 INFO:teuthology.orchestra.run.smithi181.stdout: "name": "mon.a", 2024-09-11T14:41:17.515 INFO:teuthology.orchestra.run.smithi181.stdout: "cluster": "ceph", 2024-09-11T14:41:17.515 INFO:teuthology.orchestra.run.smithi181.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-11T14:41:17.515 INFO:teuthology.orchestra.run.smithi181.stdout: "admin_socket_mode": "", 2024-09-11T14:41:17.515 INFO:teuthology.orchestra.run.smithi181.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-11T14:41:17.515 INFO:teuthology.orchestra.run.smithi181.stdout: "auth_client_required": "cephx, none", 2024-09-11T14:41:17.515 INFO:teuthology.orchestra.run.smithi181.stdout: "auth_cluster_required": "cephx", 2024-09-11T14:41:17.515 INFO:teuthology.orchestra.run.smithi181.stdout: "auth_debug": "true", 2024-09-11T14:41:17.515 INFO:teuthology.orchestra.run.smithi181.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-11T14:41:17.515 INFO:teuthology.orchestra.run.smithi181.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-11T14:41:17.515 INFO:teuthology.orchestra.run.smithi181.stdout: "auth_service_required": "cephx", 2024-09-11T14:41:17.515 INFO:teuthology.orchestra.run.smithi181.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-11T14:41:17.515 INFO:teuthology.orchestra.run.smithi181.stdout: "auth_supported": "cephx", 2024-09-11T14:41:17.515 INFO:teuthology.orchestra.run.smithi181.stdout: "bdev_aio": "true", 2024-09-11T14:41:17.516 INFO:teuthology.orchestra.run.smithi181.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-11T14:41:17.516 INFO:teuthology.orchestra.run.smithi181.stdout: "bdev_aio_poll_ms": "250", 2024-09-11T14:41:17.516 INFO:teuthology.orchestra.run.smithi181.stdout: "bdev_aio_reap_max": "16", 2024-09-11T14:41:17.516 INFO:teuthology.orchestra.run.smithi181.stdout: "bdev_async_discard_threads": "0", 2024-09-11T14:41:17.516 INFO:teuthology.orchestra.run.smithi181.stdout: "bdev_block_size": "4096", 2024-09-11T14:41:17.516 INFO:teuthology.orchestra.run.smithi181.stdout: "bdev_debug_aio": "false", 2024-09-11T14:41:17.516 INFO:teuthology.orchestra.run.smithi181.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-11T14:41:17.516 INFO:teuthology.orchestra.run.smithi181.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-11T14:41:17.516 INFO:teuthology.orchestra.run.smithi181.stdout: "bdev_debug_inflight_ios": "false", 2024-09-11T14:41:17.516 INFO:teuthology.orchestra.run.smithi181.stdout: "bdev_enable_discard": "false", 2024-09-11T14:41:17.516 INFO:teuthology.orchestra.run.smithi181.stdout: "bdev_flock_retry": "3", 2024-09-11T14:41:17.516 INFO:teuthology.orchestra.run.smithi181.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-11T14:41:17.516 INFO:teuthology.orchestra.run.smithi181.stdout: "bdev_inject_crash": "0", 2024-09-11T14:41:17.516 INFO:teuthology.orchestra.run.smithi181.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-11T14:41:17.517 INFO:teuthology.orchestra.run.smithi181.stdout: "bdev_ioring": "false", 2024-09-11T14:41:17.517 INFO:teuthology.orchestra.run.smithi181.stdout: "bdev_ioring_hipri": "false", 2024-09-11T14:41:17.517 INFO:teuthology.orchestra.run.smithi181.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-11T14:41:17.517 INFO:teuthology.orchestra.run.smithi181.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-11T14:41:17.517 INFO:teuthology.orchestra.run.smithi181.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-11T14:41:17.517 INFO:teuthology.orchestra.run.smithi181.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-11T14:41:17.517 INFO:teuthology.orchestra.run.smithi181.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-11T14:41:17.517 INFO:teuthology.orchestra.run.smithi181.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-11T14:41:17.517 INFO:teuthology.orchestra.run.smithi181.stdout: "bdev_type": "", 2024-09-11T14:41:17.517 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_alloc_size": "1048576", 2024-09-11T14:41:17.517 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_allocator": "hybrid", 2024-09-11T14:41:17.517 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_buffered_io": "true", 2024-09-11T14:41:17.517 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_check_for_zeros": "false", 2024-09-11T14:41:17.518 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-11T14:41:17.518 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-11T14:41:17.518 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_compact_log_sync": "false", 2024-09-11T14:41:17.518 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-11T14:41:17.518 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-11T14:41:17.518 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-11T14:41:17.518 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-11T14:41:17.518 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_max_log_runway": "4194304", 2024-09-11T14:41:17.518 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_max_prefetch": "1048576", 2024-09-11T14:41:17.518 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_min_flush_size": "524288", 2024-09-11T14:41:17.518 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_min_log_runway": "1048576", 2024-09-11T14:41:17.518 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_replay_recovery": "false", 2024-09-11T14:41:17.518 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-11T14:41:17.519 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-11T14:41:17.519 INFO:teuthology.orchestra.run.smithi181.stdout: "bluefs_sync_write": "false", 2024-09-11T14:41:17.519 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-11T14:41:17.519 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-11T14:41:17.519 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-11T14:41:17.519 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_allocation_from_file": "true", 2024-09-11T14:41:17.519 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_allocator": "hybrid", 2024-09-11T14:41:17.519 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_async_db_compaction": "true", 2024-09-11T14:41:17.519 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-11T14:41:17.519 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-11T14:41:17.519 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-11T14:41:17.519 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-11T14:41:17.520 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_bdev_label_multi": "true", 2024-09-11T14:41:17.520 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-11T14:41:17.520 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-11T14:41:17.520 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-11T14:41:17.520 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-11T14:41:17.520 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-11T14:41:17.520 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_block_create": "true", 2024-09-11T14:41:17.520 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_block_db_create": "false", 2024-09-11T14:41:17.520 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_block_db_path": "", 2024-09-11T14:41:17.520 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_block_db_size": "0", 2024-09-11T14:41:17.520 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_block_path": "", 2024-09-11T14:41:17.520 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_block_preallocate_file": "false", 2024-09-11T14:41:17.520 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_block_size": "107374182400", 2024-09-11T14:41:17.521 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_block_wal_create": "false", 2024-09-11T14:41:17.521 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_block_wal_path": "", 2024-09-11T14:41:17.521 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_block_wal_size": "100663296", 2024-09-11T14:41:17.521 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_bluefs": "true", 2024-09-11T14:41:17.521 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-11T14:41:17.521 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-11T14:41:17.521 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-11T14:41:17.521 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-09-11T14:41:17.521 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-11T14:41:17.521 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-11T14:41:17.521 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-11T14:41:17.521 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-11T14:41:17.521 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-11T14:41:17.522 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_autotune": "true", 2024-09-11T14:41:17.522 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-11T14:41:17.522 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-11T14:41:17.522 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-11T14:41:17.522 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-11T14:41:17.522 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_size": "0", 2024-09-11T14:41:17.522 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-11T14:41:17.522 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-11T14:41:17.522 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-11T14:41:17.522 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-11T14:41:17.522 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cache_type": "2q", 2024-09-11T14:41:17.522 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-11T14:41:17.522 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_clone_cow": "true", 2024-09-11T14:41:17.523 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-11T14:41:17.523 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-11T14:41:17.523 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-11T14:41:17.523 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-11T14:41:17.523 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-11T14:41:17.523 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-11T14:41:17.523 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-11T14:41:17.523 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_compression_mode": "none", 2024-09-11T14:41:17.523 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-11T14:41:17.523 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_csum_type": "crc32c", 2024-09-11T14:41:17.523 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-09-11T14:41:17.523 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-11T14:41:17.523 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_debug_freelist": "false", 2024-09-11T14:41:17.523 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-11T14:41:17.524 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-11T14:41:17.524 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-11T14:41:17.524 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-11T14:41:17.524 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-11T14:41:17.524 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-11T14:41:17.524 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-11T14:41:17.524 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-11T14:41:17.524 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-11T14:41:17.524 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-11T14:41:17.524 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-11T14:41:17.524 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-11T14:41:17.524 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_debug_small_allocations": "0", 2024-09-11T14:41:17.524 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-11T14:41:17.525 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_default_buffered_read": "true", 2024-09-11T14:41:17.525 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_default_buffered_write": "false", 2024-09-11T14:41:17.525 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-11T14:41:17.525 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-11T14:41:17.525 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-11T14:41:17.525 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-11T14:41:17.525 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-11T14:41:17.525 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-11T14:41:17.525 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-11T14:41:17.525 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-11T14:41:17.525 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-11T14:41:17.525 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_fail_eio": "false", 2024-09-11T14:41:17.526 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-11T14:41:17.526 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-11T14:41:17.526 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-11T14:41:17.526 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-11T14:41:17.526 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-11T14:41:17.526 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-11T14:41:17.526 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_fsck_on_mount": "false", 2024-09-11T14:41:17.526 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-11T14:41:17.526 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_fsck_on_umount": "false", 2024-09-11T14:41:17.526 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-11T14:41:17.526 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-11T14:41:17.526 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-11T14:41:17.526 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-11T14:41:17.527 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-11T14:41:17.527 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-11T14:41:17.527 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-11T14:41:17.527 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-11T14:41:17.527 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_ignore_data_csum": "false", 2024-09-11T14:41:17.527 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-11T14:41:17.527 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-11T14:41:17.527 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-11T14:41:17.527 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-11T14:41:17.527 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_log_op_age": "5.000000", 2024-09-11T14:41:17.527 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_max_alloc_size": "0", 2024-09-11T14:41:17.527 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_max_blob_size": "0", 2024-09-11T14:41:17.527 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-11T14:41:17.527 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-11T14:41:17.528 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-11T14:41:17.528 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_max_deferred_txc": "32", 2024-09-11T14:41:17.528 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_min_alloc_size": "0", 2024-09-11T14:41:17.528 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-11T14:41:17.528 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-11T14:41:17.528 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_nid_prealloc": "1024", 2024-09-11T14:41:17.528 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-11T14:41:17.528 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-11T14:41:17.528 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-11T14:41:17.528 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-11T14:41:17.528 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_retry_disk_reads": "3", 2024-09-11T14:41:17.528 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_rocksdb_cf": "true", 2024-09-11T14:41:17.528 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-11T14:41:17.529 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-11T14:41:17.529 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-11T14:41:17.529 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-11T14:41:17.529 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-11T14:41:17.529 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-11T14:41:17.529 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-11T14:41:17.529 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-11T14:41:17.530 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_spdk_mem": "512", 2024-09-11T14:41:17.530 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-11T14:41:17.530 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-11T14:41:17.530 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-11T14:41:17.530 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-11T14:41:17.530 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-11T14:41:17.530 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-11T14:41:17.530 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-11T14:41:17.530 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_tracing": "false", 2024-09-11T14:41:17.530 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-11T14:41:17.530 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-11T14:41:17.530 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-11T14:41:17.530 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-11T14:41:17.531 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-11T14:41:17.531 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-11T14:41:17.531 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-11T14:41:17.531 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-11T14:41:17.531 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-11T14:41:17.531 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_write_v2": "false", 2024-09-11T14:41:17.531 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_write_v2_random": "false", 2024-09-11T14:41:17.531 INFO:teuthology.orchestra.run.smithi181.stdout: "bluestore_zero_block_detection": "false", 2024-09-11T14:41:17.531 INFO:teuthology.orchestra.run.smithi181.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-11T14:41:17.531 INFO:teuthology.orchestra.run.smithi181.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-11T14:41:17.531 INFO:teuthology.orchestra.run.smithi181.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-11T14:41:17.531 INFO:teuthology.orchestra.run.smithi181.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-11T14:41:17.531 INFO:teuthology.orchestra.run.smithi181.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-11T14:41:17.532 INFO:teuthology.orchestra.run.smithi181.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-11T14:41:17.532 INFO:teuthology.orchestra.run.smithi181.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-11T14:41:17.532 INFO:teuthology.orchestra.run.smithi181.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-11T14:41:17.532 INFO:teuthology.orchestra.run.smithi181.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-11T14:41:17.532 INFO:teuthology.orchestra.run.smithi181.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-11T14:41:17.532 INFO:teuthology.orchestra.run.smithi181.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-11T14:41:17.532 INFO:teuthology.orchestra.run.smithi181.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-11T14:41:17.532 INFO:teuthology.orchestra.run.smithi181.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-11T14:41:17.532 INFO:teuthology.orchestra.run.smithi181.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-11T14:41:17.532 INFO:teuthology.orchestra.run.smithi181.stdout: "cephx_cluster_require_signatures": "false", 2024-09-11T14:41:17.532 INFO:teuthology.orchestra.run.smithi181.stdout: "cephx_cluster_require_version": "2", 2024-09-11T14:41:17.533 INFO:teuthology.orchestra.run.smithi181.stdout: "cephx_require_signatures": "false", 2024-09-11T14:41:17.533 INFO:teuthology.orchestra.run.smithi181.stdout: "cephx_require_version": "2", 2024-09-11T14:41:17.533 INFO:teuthology.orchestra.run.smithi181.stdout: "cephx_service_require_signatures": "false", 2024-09-11T14:41:17.533 INFO:teuthology.orchestra.run.smithi181.stdout: "cephx_service_require_version": "2", 2024-09-11T14:41:17.533 INFO:teuthology.orchestra.run.smithi181.stdout: "cephx_sign_messages": "true", 2024-09-11T14:41:17.533 INFO:teuthology.orchestra.run.smithi181.stdout: "chdir": "", 2024-09-11T14:41:17.533 INFO:teuthology.orchestra.run.smithi181.stdout: "client_acl_type": "", 2024-09-11T14:41:17.533 INFO:teuthology.orchestra.run.smithi181.stdout: "client_asio_thread_count": "2", 2024-09-11T14:41:17.533 INFO:teuthology.orchestra.run.smithi181.stdout: "client_cache_mid": "0.750000", 2024-09-11T14:41:17.533 INFO:teuthology.orchestra.run.smithi181.stdout: "client_cache_size": "16384", 2024-09-11T14:41:17.533 INFO:teuthology.orchestra.run.smithi181.stdout: "client_caps_release_delay": "5", 2024-09-11T14:41:17.533 INFO:teuthology.orchestra.run.smithi181.stdout: "client_check_pool_perm": "true", 2024-09-11T14:41:17.533 INFO:teuthology.orchestra.run.smithi181.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-11T14:41:17.534 INFO:teuthology.orchestra.run.smithi181.stdout: "client_debug_force_sync_read": "false", 2024-09-11T14:41:17.534 INFO:teuthology.orchestra.run.smithi181.stdout: "client_debug_getattr_caps": "false", 2024-09-11T14:41:17.534 INFO:teuthology.orchestra.run.smithi181.stdout: "client_debug_inject_features": "", 2024-09-11T14:41:17.534 INFO:teuthology.orchestra.run.smithi181.stdout: "client_debug_inject_tick_delay": "0", 2024-09-11T14:41:17.534 INFO:teuthology.orchestra.run.smithi181.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-11T14:41:17.534 INFO:teuthology.orchestra.run.smithi181.stdout: "client_die_on_failed_remount": "false", 2024-09-11T14:41:17.534 INFO:teuthology.orchestra.run.smithi181.stdout: "client_dirsize_rbytes": "true", 2024-09-11T14:41:17.534 INFO:teuthology.orchestra.run.smithi181.stdout: "client_force_lazyio": "false", 2024-09-11T14:41:17.534 INFO:teuthology.orchestra.run.smithi181.stdout: "client_fs": "", 2024-09-11T14:41:17.534 INFO:teuthology.orchestra.run.smithi181.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-11T14:41:17.534 INFO:teuthology.orchestra.run.smithi181.stdout: "client_inject_release_failure": "false", 2024-09-11T14:41:17.534 INFO:teuthology.orchestra.run.smithi181.stdout: "client_max_inline_size": "4096", 2024-09-11T14:41:17.534 INFO:teuthology.orchestra.run.smithi181.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-11T14:41:17.534 INFO:teuthology.orchestra.run.smithi181.stdout: "client_mds_namespace": "", 2024-09-11T14:41:17.535 INFO:teuthology.orchestra.run.smithi181.stdout: "client_metadata": "", 2024-09-11T14:41:17.535 INFO:teuthology.orchestra.run.smithi181.stdout: "client_mount_gid": "-1", 2024-09-11T14:41:17.535 INFO:teuthology.orchestra.run.smithi181.stdout: "client_mount_timeout": "300", 2024-09-11T14:41:17.535 INFO:teuthology.orchestra.run.smithi181.stdout: "client_mount_uid": "-1", 2024-09-11T14:41:17.535 INFO:teuthology.orchestra.run.smithi181.stdout: "client_mountpoint": "/", 2024-09-11T14:41:17.535 INFO:teuthology.orchestra.run.smithi181.stdout: "client_notify_timeout": "10", 2024-09-11T14:41:17.535 INFO:teuthology.orchestra.run.smithi181.stdout: "client_oc": "true", 2024-09-11T14:41:17.535 INFO:teuthology.orchestra.run.smithi181.stdout: "client_oc_max_dirty": "104857600", 2024-09-11T14:41:17.535 INFO:teuthology.orchestra.run.smithi181.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-11T14:41:17.535 INFO:teuthology.orchestra.run.smithi181.stdout: "client_oc_max_objects": "1000", 2024-09-11T14:41:17.535 INFO:teuthology.orchestra.run.smithi181.stdout: "client_oc_size": "209715200", 2024-09-11T14:41:17.535 INFO:teuthology.orchestra.run.smithi181.stdout: "client_oc_target_dirty": "8388608", 2024-09-11T14:41:17.535 INFO:teuthology.orchestra.run.smithi181.stdout: "client_permissions": "true", 2024-09-11T14:41:17.536 INFO:teuthology.orchestra.run.smithi181.stdout: "client_quota": "true", 2024-09-11T14:41:17.536 INFO:teuthology.orchestra.run.smithi181.stdout: "client_quota_df": "true", 2024-09-11T14:41:17.536 INFO:teuthology.orchestra.run.smithi181.stdout: "client_readahead_max_bytes": "0", 2024-09-11T14:41:17.536 INFO:teuthology.orchestra.run.smithi181.stdout: "client_readahead_max_periods": "4", 2024-09-11T14:41:17.536 INFO:teuthology.orchestra.run.smithi181.stdout: "client_readahead_min": "131072", 2024-09-11T14:41:17.536 INFO:teuthology.orchestra.run.smithi181.stdout: "client_reconnect_stale": "false", 2024-09-11T14:41:17.536 INFO:teuthology.orchestra.run.smithi181.stdout: "client_shutdown_timeout": "30", 2024-09-11T14:41:17.536 INFO:teuthology.orchestra.run.smithi181.stdout: "client_snapdir": ".snap", 2024-09-11T14:41:17.536 INFO:teuthology.orchestra.run.smithi181.stdout: "client_tick_interval": "1", 2024-09-11T14:41:17.536 INFO:teuthology.orchestra.run.smithi181.stdout: "client_trace": "", 2024-09-11T14:41:17.536 INFO:teuthology.orchestra.run.smithi181.stdout: "client_try_dentry_invalidate": "false", 2024-09-11T14:41:17.536 INFO:teuthology.orchestra.run.smithi181.stdout: "client_use_faked_inos": "false", 2024-09-11T14:41:17.537 INFO:teuthology.orchestra.run.smithi181.stdout: "client_use_random_mds": "false", 2024-09-11T14:41:17.537 INFO:teuthology.orchestra.run.smithi181.stdout: "clog_to_graylog": "false", 2024-09-11T14:41:17.537 INFO:teuthology.orchestra.run.smithi181.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-11T14:41:17.537 INFO:teuthology.orchestra.run.smithi181.stdout: "clog_to_graylog_port": "12201", 2024-09-11T14:41:17.537 INFO:teuthology.orchestra.run.smithi181.stdout: "clog_to_monitors": "default=true", 2024-09-11T14:41:17.537 INFO:teuthology.orchestra.run.smithi181.stdout: "clog_to_syslog": "false", 2024-09-11T14:41:17.537 INFO:teuthology.orchestra.run.smithi181.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-11T14:41:17.537 INFO:teuthology.orchestra.run.smithi181.stdout: "clog_to_syslog_level": "info", 2024-09-11T14:41:17.537 INFO:teuthology.orchestra.run.smithi181.stdout: "cluster_addr": "-", 2024-09-11T14:41:17.537 INFO:teuthology.orchestra.run.smithi181.stdout: "cluster_network": "", 2024-09-11T14:41:17.537 INFO:teuthology.orchestra.run.smithi181.stdout: "cluster_network_interface": "", 2024-09-11T14:41:17.537 INFO:teuthology.orchestra.run.smithi181.stdout: "compressor_zlib_isal": "false", 2024-09-11T14:41:17.537 INFO:teuthology.orchestra.run.smithi181.stdout: "compressor_zlib_level": "5", 2024-09-11T14:41:17.538 INFO:teuthology.orchestra.run.smithi181.stdout: "compressor_zlib_winsize": "-15", 2024-09-11T14:41:17.538 INFO:teuthology.orchestra.run.smithi181.stdout: "compressor_zstd_level": "1", 2024-09-11T14:41:17.538 INFO:teuthology.orchestra.run.smithi181.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-09-11T14:41:17.538 INFO:teuthology.orchestra.run.smithi181.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-11T14:41:17.538 INFO:teuthology.orchestra.run.smithi181.stdout: "crimson_alien_op_num_threads": "6", 2024-09-11T14:41:17.538 INFO:teuthology.orchestra.run.smithi181.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-11T14:41:17.538 INFO:teuthology.orchestra.run.smithi181.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-11T14:41:17.538 INFO:teuthology.orchestra.run.smithi181.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-11T14:41:17.538 INFO:teuthology.orchestra.run.smithi181.stdout: "crimson_osd_stat_interval": "0", 2024-09-11T14:41:17.538 INFO:teuthology.orchestra.run.smithi181.stdout: "crimson_seastar_cpu_cores": "", 2024-09-11T14:41:17.538 INFO:teuthology.orchestra.run.smithi181.stdout: "crimson_seastar_num_threads": "0", 2024-09-11T14:41:17.538 INFO:teuthology.orchestra.run.smithi181.stdout: "crush_location": "", 2024-09-11T14:41:17.538 INFO:teuthology.orchestra.run.smithi181.stdout: "crush_location_hook": "", 2024-09-11T14:41:17.538 INFO:teuthology.orchestra.run.smithi181.stdout: "crush_location_hook_timeout": "10", 2024-09-11T14:41:17.539 INFO:teuthology.orchestra.run.smithi181.stdout: "daemonize": "false", 2024-09-11T14:41:17.539 INFO:teuthology.orchestra.run.smithi181.stdout: "daos_pool": "tank", 2024-09-11T14:41:17.539 INFO:teuthology.orchestra.run.smithi181.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-11T14:41:17.539 INFO:teuthology.orchestra.run.smithi181.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-11T14:41:17.539 INFO:teuthology.orchestra.run.smithi181.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-11T14:41:17.539 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_alienstore": "0/5", 2024-09-11T14:41:17.539 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_allow_any_pool_priority": "false", 2024-09-11T14:41:17.539 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_asok": "1/5", 2024-09-11T14:41:17.539 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_asok_assert_abort": "false", 2024-09-11T14:41:17.539 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_asserts_on_shutdown": "true", 2024-09-11T14:41:17.539 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_auth": "20/20", 2024-09-11T14:41:17.539 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_bdev": "1/3", 2024-09-11T14:41:17.539 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_bluefs": "1/5", 2024-09-11T14:41:17.540 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_bluestore": "1/5", 2024-09-11T14:41:17.540 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_buffer": "0/1", 2024-09-11T14:41:17.540 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_ceph_dedup": "0/5", 2024-09-11T14:41:17.540 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_ceph_exporter": "1/5", 2024-09-11T14:41:17.540 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_cephfs_mirror": "0/5", 2024-09-11T14:41:17.540 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_cephsqlite": "0/5", 2024-09-11T14:41:17.540 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_client": "0/5", 2024-09-11T14:41:17.540 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_compressor": "1/5", 2024-09-11T14:41:17.540 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_context": "0/1", 2024-09-11T14:41:17.540 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_crimson_interrupt": "0/5", 2024-09-11T14:41:17.540 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_crush": "1/1", 2024-09-11T14:41:17.540 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_crypto": "1/5", 2024-09-11T14:41:17.541 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_cyanstore": "0/5", 2024-09-11T14:41:17.541 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_deliberately_leak_memory": "false", 2024-09-11T14:41:17.541 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_disable_randomized_ping": "false", 2024-09-11T14:41:17.541 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_dpdk": "1/5", 2024-09-11T14:41:17.541 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_eventtrace": "1/5", 2024-09-11T14:41:17.541 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_filer": "0/1", 2024-09-11T14:41:17.541 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_filestore": "1/3", 2024-09-11T14:41:17.541 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_finisher": "1/1", 2024-09-11T14:41:17.541 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_fuse": "1/5", 2024-09-11T14:41:17.541 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_heartbeat_testing_span": "0", 2024-09-11T14:41:17.541 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_heartbeatmap": "1/5", 2024-09-11T14:41:17.541 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-11T14:41:17.541 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_javaclient": "1/5", 2024-09-11T14:41:17.541 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_journal": "1/3", 2024-09-11T14:41:17.542 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_journaler": "0/5", 2024-09-11T14:41:17.542 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_kstore": "1/5", 2024-09-11T14:41:17.542 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_lockdep": "0/1", 2024-09-11T14:41:17.542 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_mclock": "1/5", 2024-09-11T14:41:17.542 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_mds": "1/5", 2024-09-11T14:41:17.542 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_mds_balancer": "1/5", 2024-09-11T14:41:17.542 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_mds_locker": "1/5", 2024-09-11T14:41:17.542 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_mds_log": "1/5", 2024-09-11T14:41:17.542 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_mds_log_expire": "1/5", 2024-09-11T14:41:17.542 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_mds_migrator": "1/5", 2024-09-11T14:41:17.542 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_mds_quiesce": "3/5", 2024-09-11T14:41:17.542 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_memstore": "1/5", 2024-09-11T14:41:17.542 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_mgr": "2/5", 2024-09-11T14:41:17.542 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_mgrc": "1/5", 2024-09-11T14:41:17.543 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_mon": "20/20", 2024-09-11T14:41:17.543 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_monc": "0/10", 2024-09-11T14:41:17.543 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_ms": "1/1", 2024-09-11T14:41:17.543 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_none": "0/5", 2024-09-11T14:41:17.543 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_objclass": "0/5", 2024-09-11T14:41:17.543 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_objectcacher": "0/5", 2024-09-11T14:41:17.543 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_objecter": "0/1", 2024-09-11T14:41:17.543 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_optracker": "0/5", 2024-09-11T14:41:17.543 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_osd": "1/5", 2024-09-11T14:41:17.543 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_paxos": "20/20", 2024-09-11T14:41:17.543 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_perfcounter": "1/5", 2024-09-11T14:41:17.543 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_prioritycache": "1/5", 2024-09-11T14:41:17.543 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_rados": "0/5", 2024-09-11T14:41:17.543 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_rbd": "0/5", 2024-09-11T14:41:17.544 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_rbd_mirror": "0/5", 2024-09-11T14:41:17.544 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_rbd_pwl": "0/5", 2024-09-11T14:41:17.544 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_rbd_replay": "0/5", 2024-09-11T14:41:17.544 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_refs": "0/0", 2024-09-11T14:41:17.544 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_reserver": "1/1", 2024-09-11T14:41:17.544 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_rgw": "1/5", 2024-09-11T14:41:17.544 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_rgw_access": "1/5", 2024-09-11T14:41:17.544 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_rgw_datacache": "1/5", 2024-09-11T14:41:17.544 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_rgw_dbstore": "1/5", 2024-09-11T14:41:17.544 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_rgw_flight": "1/5", 2024-09-11T14:41:17.544 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-11T14:41:17.544 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_rgw_notification": "1/5", 2024-09-11T14:41:17.544 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_rgw_sync": "1/5", 2024-09-11T14:41:17.545 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_rocksdb": "4/5", 2024-09-11T14:41:17.545 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_seastore": "0/5", 2024-09-11T14:41:17.545 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_seastore_backref": "0/5", 2024-09-11T14:41:17.545 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_seastore_cache": "0/5", 2024-09-11T14:41:17.545 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_seastore_cleaner": "0/5", 2024-09-11T14:41:17.545 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_seastore_device": "0/5", 2024-09-11T14:41:17.545 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_seastore_epm": "0/5", 2024-09-11T14:41:17.545 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-11T14:41:17.545 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_seastore_journal": "0/5", 2024-09-11T14:41:17.545 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_seastore_lba": "0/5", 2024-09-11T14:41:17.545 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_seastore_odata": "0/5", 2024-09-11T14:41:17.545 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_seastore_omap": "0/5", 2024-09-11T14:41:17.545 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_seastore_onode": "0/5", 2024-09-11T14:41:17.545 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_seastore_t": "0/5", 2024-09-11T14:41:17.546 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_seastore_tm": "0/5", 2024-09-11T14:41:17.546 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_striper": "0/1", 2024-09-11T14:41:17.546 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_test": "0/5", 2024-09-11T14:41:17.546 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_throttle": "1/1", 2024-09-11T14:41:17.546 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_timer": "0/1", 2024-09-11T14:41:17.546 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_tp": "0/5", 2024-09-11T14:41:17.546 INFO:teuthology.orchestra.run.smithi181.stdout: "debug_trace": "1/5", 2024-09-11T14:41:17.546 INFO:teuthology.orchestra.run.smithi181.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-11T14:41:17.546 INFO:teuthology.orchestra.run.smithi181.stdout: "device_failure_prediction_mode": "none", 2024-09-11T14:41:17.546 INFO:teuthology.orchestra.run.smithi181.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "crimson", 2024-09-11T14:41:17.546 INFO:teuthology.orchestra.run.smithi181.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-11T14:41:17.546 INFO:teuthology.orchestra.run.smithi181.stdout: "err_to_graylog": "false", 2024-09-11T14:41:17.546 INFO:teuthology.orchestra.run.smithi181.stdout: "err_to_journald": "false", 2024-09-11T14:41:17.547 INFO:teuthology.orchestra.run.smithi181.stdout: "err_to_stderr": "true", 2024-09-11T14:41:17.547 INFO:teuthology.orchestra.run.smithi181.stdout: "err_to_syslog": "false", 2024-09-11T14:41:17.547 INFO:teuthology.orchestra.run.smithi181.stdout: "event_tracing": "false", 2024-09-11T14:41:17.547 INFO:teuthology.orchestra.run.smithi181.stdout: "exporter_addr": "0.0.0.0", 2024-09-11T14:41:17.547 INFO:teuthology.orchestra.run.smithi181.stdout: "exporter_cert_file": "", 2024-09-11T14:41:17.547 INFO:teuthology.orchestra.run.smithi181.stdout: "exporter_http_port": "9926", 2024-09-11T14:41:17.547 INFO:teuthology.orchestra.run.smithi181.stdout: "exporter_key_file": "", 2024-09-11T14:41:17.547 INFO:teuthology.orchestra.run.smithi181.stdout: "exporter_prio_limit": "5", 2024-09-11T14:41:17.547 INFO:teuthology.orchestra.run.smithi181.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-11T14:41:17.547 INFO:teuthology.orchestra.run.smithi181.stdout: "exporter_sort_metrics": "true", 2024-09-11T14:41:17.547 INFO:teuthology.orchestra.run.smithi181.stdout: "exporter_stats_period": "5", 2024-09-11T14:41:17.547 INFO:teuthology.orchestra.run.smithi181.stdout: "fake_statfs_for_testing": "0", 2024-09-11T14:41:17.547 INFO:teuthology.orchestra.run.smithi181.stdout: "fatal_signal_handlers": "true", 2024-09-11T14:41:17.547 INFO:teuthology.orchestra.run.smithi181.stdout: "filer_max_purge_ops": "10", 2024-09-11T14:41:17.548 INFO:teuthology.orchestra.run.smithi181.stdout: "filer_max_truncate_ops": "128", 2024-09-11T14:41:17.548 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_apply_finisher_threads": "1", 2024-09-11T14:41:17.548 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_blackhole": "false", 2024-09-11T14:41:17.548 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_btrfs_clone_range": "true", 2024-09-11T14:41:17.548 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_btrfs_snap": "true", 2024-09-11T14:41:17.548 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_caller_concurrency": "10", 2024-09-11T14:41:17.548 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_collect_device_partition_information": "true", 2024-09-11T14:41:17.548 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_commit_timeout": "600.000000", 2024-09-11T14:41:17.548 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_debug_inject_read_err": "false", 2024-09-11T14:41:17.548 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_debug_omap_check": "false", 2024-09-11T14:41:17.548 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_debug_verify_split": "false", 2024-09-11T14:41:17.548 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_dump_file": "", 2024-09-11T14:41:17.548 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-11T14:41:17.549 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-11T14:41:17.549 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_fadvise": "true", 2024-09-11T14:41:17.549 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_fail_eio": "true", 2024-09-11T14:41:17.549 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_fd_cache_shards": "16", 2024-09-11T14:41:17.549 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_fd_cache_size": "128", 2024-09-11T14:41:17.549 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_fiemap": "false", 2024-09-11T14:41:17.549 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_fiemap_threshold": "4096", 2024-09-11T14:41:17.549 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-11T14:41:17.549 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-11T14:41:17.549 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_inject_stall": "0", 2024-09-11T14:41:17.549 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_journal_parallel": "false", 2024-09-11T14:41:17.549 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_journal_trailing": "false", 2024-09-11T14:41:17.549 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_journal_writeahead": "false", 2024-09-11T14:41:17.550 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_kill_at": "0", 2024-09-11T14:41:17.550 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-11T14:41:17.550 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-11T14:41:17.550 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-11T14:41:17.550 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-11T14:41:17.550 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-11T14:41:17.550 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_max_inline_xattrs": "0", 2024-09-11T14:41:17.550 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-11T14:41:17.550 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-11T14:41:17.550 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-11T14:41:17.550 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-11T14:41:17.550 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_max_xattr_value_size": "0", 2024-09-11T14:41:17.550 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-11T14:41:17.551 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-11T14:41:17.551 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-11T14:41:17.551 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_merge_threshold": "-10", 2024-09-11T14:41:17.551 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-11T14:41:17.551 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_odsync_write": "false", 2024-09-11T14:41:17.551 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_omap_backend": "rocksdb", 2024-09-11T14:41:17.551 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_omap_backend_path": "", 2024-09-11T14:41:17.551 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-11T14:41:17.551 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-11T14:41:17.551 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-11T14:41:17.551 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_op_thread_timeout": "60", 2024-09-11T14:41:17.551 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_op_threads": "2", 2024-09-11T14:41:17.551 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_punch_hole": "false", 2024-09-11T14:41:17.551 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-11T14:41:17.552 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-11T14:41:17.552 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-11T14:41:17.552 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-11T14:41:17.552 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-11T14:41:17.552 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-11T14:41:17.552 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-11T14:41:17.552 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-11T14:41:17.552 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-11T14:41:17.552 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_queue_max_ops": "50", 2024-09-11T14:41:17.552 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-11T14:41:17.552 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_seek_data_hole": "false", 2024-09-11T14:41:17.552 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_sloppy_crc": "false", 2024-09-11T14:41:17.552 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-11T14:41:17.553 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_splice": "false", 2024-09-11T14:41:17.553 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_split_multiple": "2", 2024-09-11T14:41:17.553 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_split_rand_factor": "20", 2024-09-11T14:41:17.553 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_update_to": "1000", 2024-09-11T14:41:17.553 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-11T14:41:17.553 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-11T14:41:17.553 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-11T14:41:17.553 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-11T14:41:17.553 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-11T14:41:17.553 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-11T14:41:17.553 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_wbthrottle_enable": "true", 2024-09-11T14:41:17.553 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-11T14:41:17.553 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-11T14:41:17.554 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-11T14:41:17.554 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-11T14:41:17.554 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-11T14:41:17.554 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-11T14:41:17.554 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_xfs_extsize": "false", 2024-09-11T14:41:17.554 INFO:teuthology.orchestra.run.smithi181.stdout: "filestore_zfs_snap": "false", 2024-09-11T14:41:17.554 INFO:teuthology.orchestra.run.smithi181.stdout: "fio_dir": "/tmp/fio", 2024-09-11T14:41:17.554 INFO:teuthology.orchestra.run.smithi181.stdout: "fsid": "0f7e90de-623e-43a2-a836-b493e06f03fc", 2024-09-11T14:41:17.554 INFO:teuthology.orchestra.run.smithi181.stdout: "fuse_allow_other": "true", 2024-09-11T14:41:17.554 INFO:teuthology.orchestra.run.smithi181.stdout: "fuse_atomic_o_trunc": "true", 2024-09-11T14:41:17.554 INFO:teuthology.orchestra.run.smithi181.stdout: "fuse_big_writes": "true", 2024-09-11T14:41:17.554 INFO:teuthology.orchestra.run.smithi181.stdout: "fuse_debug": "false", 2024-09-11T14:41:17.554 INFO:teuthology.orchestra.run.smithi181.stdout: "fuse_default_permissions": "false", 2024-09-11T14:41:17.555 INFO:teuthology.orchestra.run.smithi181.stdout: "fuse_disable_pagecache": "false", 2024-09-11T14:41:17.555 INFO:teuthology.orchestra.run.smithi181.stdout: "fuse_max_write": "0", 2024-09-11T14:41:17.555 INFO:teuthology.orchestra.run.smithi181.stdout: "fuse_multithreaded": "true", 2024-09-11T14:41:17.555 INFO:teuthology.orchestra.run.smithi181.stdout: "fuse_require_active_mds": "true", 2024-09-11T14:41:17.555 INFO:teuthology.orchestra.run.smithi181.stdout: "fuse_set_user_groups": "true", 2024-09-11T14:41:17.555 INFO:teuthology.orchestra.run.smithi181.stdout: "fuse_splice_move": "true", 2024-09-11T14:41:17.555 INFO:teuthology.orchestra.run.smithi181.stdout: "fuse_splice_read": "true", 2024-09-11T14:41:17.555 INFO:teuthology.orchestra.run.smithi181.stdout: "fuse_splice_write": "true", 2024-09-11T14:41:17.555 INFO:teuthology.orchestra.run.smithi181.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-11T14:41:17.555 INFO:teuthology.orchestra.run.smithi181.stdout: "fuse_use_invalidate_cb": "true", 2024-09-11T14:41:17.555 INFO:teuthology.orchestra.run.smithi181.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-11T14:41:17.555 INFO:teuthology.orchestra.run.smithi181.stdout: "gss_target_name": "ceph", 2024-09-11T14:41:17.555 INFO:teuthology.orchestra.run.smithi181.stdout: "heartbeat_file": "", 2024-09-11T14:41:17.556 INFO:teuthology.orchestra.run.smithi181.stdout: "heartbeat_inject_failure": "0", 2024-09-11T14:41:17.556 INFO:teuthology.orchestra.run.smithi181.stdout: "heartbeat_interval": "5", 2024-09-11T14:41:17.556 INFO:teuthology.orchestra.run.smithi181.stdout: "host": "smithi181", 2024-09-11T14:41:17.556 INFO:teuthology.orchestra.run.smithi181.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-11T14:41:17.556 INFO:teuthology.orchestra.run.smithi181.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-11T14:41:17.556 INFO:teuthology.orchestra.run.smithi181.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-11T14:41:17.556 INFO:teuthology.orchestra.run.smithi181.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-11T14:41:17.556 INFO:teuthology.orchestra.run.smithi181.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-11T14:41:17.556 INFO:teuthology.orchestra.run.smithi181.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-11T14:41:17.556 INFO:teuthology.orchestra.run.smithi181.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-11T14:41:17.556 INFO:teuthology.orchestra.run.smithi181.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-11T14:41:17.556 INFO:teuthology.orchestra.run.smithi181.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-11T14:41:17.556 INFO:teuthology.orchestra.run.smithi181.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-11T14:41:17.557 INFO:teuthology.orchestra.run.smithi181.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-11T14:41:17.557 INFO:teuthology.orchestra.run.smithi181.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-11T14:41:17.557 INFO:teuthology.orchestra.run.smithi181.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-11T14:41:17.557 INFO:teuthology.orchestra.run.smithi181.stdout: "inject_early_sigterm": "false", 2024-09-11T14:41:17.557 INFO:teuthology.orchestra.run.smithi181.stdout: "jaeger_agent_port": "6799", 2024-09-11T14:41:17.557 INFO:teuthology.orchestra.run.smithi181.stdout: "jaeger_tracing_enable": "false", 2024-09-11T14:41:17.557 INFO:teuthology.orchestra.run.smithi181.stdout: "journal_aio": "true", 2024-09-11T14:41:17.557 INFO:teuthology.orchestra.run.smithi181.stdout: "journal_align_min_size": "65536", 2024-09-11T14:41:17.557 INFO:teuthology.orchestra.run.smithi181.stdout: "journal_block_align": "true", 2024-09-11T14:41:17.557 INFO:teuthology.orchestra.run.smithi181.stdout: "journal_block_size": "4096", 2024-09-11T14:41:17.557 INFO:teuthology.orchestra.run.smithi181.stdout: "journal_dio": "true", 2024-09-11T14:41:17.557 INFO:teuthology.orchestra.run.smithi181.stdout: "journal_discard": "false", 2024-09-11T14:41:17.557 INFO:teuthology.orchestra.run.smithi181.stdout: "journal_force_aio": "false", 2024-09-11T14:41:17.557 INFO:teuthology.orchestra.run.smithi181.stdout: "journal_ignore_corruption": "false", 2024-09-11T14:41:17.558 INFO:teuthology.orchestra.run.smithi181.stdout: "journal_max_write_bytes": "10485760", 2024-09-11T14:41:17.558 INFO:teuthology.orchestra.run.smithi181.stdout: "journal_max_write_entries": "100", 2024-09-11T14:41:17.558 INFO:teuthology.orchestra.run.smithi181.stdout: "journal_replay_from": "0", 2024-09-11T14:41:17.558 INFO:teuthology.orchestra.run.smithi181.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-11T14:41:17.558 INFO:teuthology.orchestra.run.smithi181.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-11T14:41:17.558 INFO:teuthology.orchestra.run.smithi181.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-11T14:41:17.558 INFO:teuthology.orchestra.run.smithi181.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-11T14:41:17.558 INFO:teuthology.orchestra.run.smithi181.stdout: "journal_write_header_frequency": "0", 2024-09-11T14:41:17.558 INFO:teuthology.orchestra.run.smithi181.stdout: "journal_zero_on_create": "false", 2024-09-11T14:41:17.558 INFO:teuthology.orchestra.run.smithi181.stdout: "journaler_prefetch_periods": "10", 2024-09-11T14:41:17.558 INFO:teuthology.orchestra.run.smithi181.stdout: "journaler_prezero_periods": "5", 2024-09-11T14:41:17.558 INFO:teuthology.orchestra.run.smithi181.stdout: "journaler_write_head_interval": "15", 2024-09-11T14:41:17.558 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-11T14:41:17.558 INFO:teuthology.orchestra.run.smithi181.stdout: "keyfile": "", 2024-09-11T14:41:17.559 INFO:teuthology.orchestra.run.smithi181.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-11T14:41:17.559 INFO:teuthology.orchestra.run.smithi181.stdout: "kstore_backend": "rocksdb", 2024-09-11T14:41:17.559 INFO:teuthology.orchestra.run.smithi181.stdout: "kstore_default_stripe_size": "65536", 2024-09-11T14:41:17.559 INFO:teuthology.orchestra.run.smithi181.stdout: "kstore_fsck_on_mount": "false", 2024-09-11T14:41:17.559 INFO:teuthology.orchestra.run.smithi181.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-11T14:41:17.559 INFO:teuthology.orchestra.run.smithi181.stdout: "kstore_max_bytes": "67108864", 2024-09-11T14:41:17.559 INFO:teuthology.orchestra.run.smithi181.stdout: "kstore_max_ops": "512", 2024-09-11T14:41:17.559 INFO:teuthology.orchestra.run.smithi181.stdout: "kstore_nid_prealloc": "1024", 2024-09-11T14:41:17.559 INFO:teuthology.orchestra.run.smithi181.stdout: "kstore_onode_map_size": "1024", 2024-09-11T14:41:17.559 INFO:teuthology.orchestra.run.smithi181.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-11T14:41:17.559 INFO:teuthology.orchestra.run.smithi181.stdout: "kstore_sync_submit_transaction": "false", 2024-09-11T14:41:17.559 INFO:teuthology.orchestra.run.smithi181.stdout: "kstore_sync_transaction": "false", 2024-09-11T14:41:17.560 INFO:teuthology.orchestra.run.smithi181.stdout: "librados_thread_count": "2", 2024-09-11T14:41:17.560 INFO:teuthology.orchestra.run.smithi181.stdout: "lockdep": "false", 2024-09-11T14:41:17.560 INFO:teuthology.orchestra.run.smithi181.stdout: "lockdep_force_backtrace": "false", 2024-09-11T14:41:17.560 INFO:teuthology.orchestra.run.smithi181.stdout: "log_coarse_timestamps": "true", 2024-09-11T14:41:17.560 INFO:teuthology.orchestra.run.smithi181.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-11T14:41:17.560 INFO:teuthology.orchestra.run.smithi181.stdout: "log_flush_on_exit": "false", 2024-09-11T14:41:17.560 INFO:teuthology.orchestra.run.smithi181.stdout: "log_graylog_host": "127.0.0.1", 2024-09-11T14:41:17.560 INFO:teuthology.orchestra.run.smithi181.stdout: "log_graylog_port": "12201", 2024-09-11T14:41:17.560 INFO:teuthology.orchestra.run.smithi181.stdout: "log_max_new": "1000", 2024-09-11T14:41:17.560 INFO:teuthology.orchestra.run.smithi181.stdout: "log_max_recent": "10000", 2024-09-11T14:41:17.560 INFO:teuthology.orchestra.run.smithi181.stdout: "log_stderr_prefix": "", 2024-09-11T14:41:17.561 INFO:teuthology.orchestra.run.smithi181.stdout: "log_stop_at_utilization": "0.970000", 2024-09-11T14:41:17.561 INFO:teuthology.orchestra.run.smithi181.stdout: "log_to_file": "true", 2024-09-11T14:41:17.561 INFO:teuthology.orchestra.run.smithi181.stdout: "log_to_graylog": "false", 2024-09-11T14:41:17.561 INFO:teuthology.orchestra.run.smithi181.stdout: "log_to_journald": "false", 2024-09-11T14:41:17.561 INFO:teuthology.orchestra.run.smithi181.stdout: "log_to_stderr": "false", 2024-09-11T14:41:17.561 INFO:teuthology.orchestra.run.smithi181.stdout: "log_to_syslog": "false", 2024-09-11T14:41:17.561 INFO:teuthology.orchestra.run.smithi181.stdout: "max_rotating_auth_attempts": "10", 2024-09-11T14:41:17.561 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-11T14:41:17.561 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_action_on_write_error": "1", 2024-09-11T14:41:17.561 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_alternate_name_max": "8192", 2024-09-11T14:41:17.561 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_asio_thread_count": "2", 2024-09-11T14:41:17.561 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_bal_export_pin": "true", 2024-09-11T14:41:17.561 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_bal_fragment_dirs": "true", 2024-09-11T14:41:17.561 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-11T14:41:17.562 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_bal_fragment_interval": "5", 2024-09-11T14:41:17.562 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-11T14:41:17.562 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-11T14:41:17.562 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_bal_interval": "10", 2024-09-11T14:41:17.562 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_bal_max": "-1", 2024-09-11T14:41:17.562 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_bal_max_until": "-1", 2024-09-11T14:41:17.562 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_bal_merge_size": "50", 2024-09-11T14:41:17.562 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_bal_midchunk": "0.300000", 2024-09-11T14:41:17.562 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-11T14:41:17.562 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_bal_min_start": "0.200000", 2024-09-11T14:41:17.562 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_bal_minchunk": "0.001000", 2024-09-11T14:41:17.562 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_bal_mode": "0", 2024-09-11T14:41:17.562 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_bal_need_max": "1.200000", 2024-09-11T14:41:17.562 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_bal_need_min": "0.800000", 2024-09-11T14:41:17.563 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_bal_overload_epochs": "2", 2024-09-11T14:41:17.563 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-11T14:41:17.563 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-11T14:41:17.563 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_bal_split_bits": "3", 2024-09-11T14:41:17.563 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-11T14:41:17.563 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_bal_split_size": "10000", 2024-09-11T14:41:17.563 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-11T14:41:17.563 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_bal_target_decay": "10.000000", 2024-09-11T14:41:17.563 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-11T14:41:17.563 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_beacon_grace": "15.000000", 2024-09-11T14:41:17.563 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_beacon_interval": "4.000000", 2024-09-11T14:41:17.563 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-11T14:41:17.563 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-11T14:41:17.564 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_cache_mid": "0.700000", 2024-09-11T14:41:17.564 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-11T14:41:17.564 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_cache_quiesce_delay": "0", 2024-09-11T14:41:17.564 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-11T14:41:17.564 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-11T14:41:17.564 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-11T14:41:17.564 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_cache_release_free_interval": "10", 2024-09-11T14:41:17.564 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_cache_reservation": "0.050000", 2024-09-11T14:41:17.564 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-11T14:41:17.564 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_cache_trim_interval": "1", 2024-09-11T14:41:17.564 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_cache_trim_threshold": "262144", 2024-09-11T14:41:17.564 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-11T14:41:17.565 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-11T14:41:17.565 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-11T14:41:17.565 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_client_prealloc_inos": "1000", 2024-09-11T14:41:17.565 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-11T14:41:17.565 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_connect_bootstrapping": "false", 2024-09-11T14:41:17.565 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_damage_table_max_entries": "10000", 2024-09-11T14:41:17.565 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-11T14:41:17.565 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_debug_auth_pins": "false", 2024-09-11T14:41:17.565 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_debug_frag": "false", 2024-09-11T14:41:17.565 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_debug_scatterstat": "false", 2024-09-11T14:41:17.565 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_debug_subtrees": "false", 2024-09-11T14:41:17.565 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_decay_halflife": "5.000000", 2024-09-11T14:41:17.565 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_default_dir_hash": "2", 2024-09-11T14:41:17.566 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_defer_session_stale": "true", 2024-09-11T14:41:17.566 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_deny_all_reconnect": "false", 2024-09-11T14:41:17.566 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_dir_keys_per_op": "16384", 2024-09-11T14:41:17.566 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_dir_max_commit_size": "10", 2024-09-11T14:41:17.566 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_dir_max_entries": "0", 2024-09-11T14:41:17.566 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_dir_prefetch": "true", 2024-09-11T14:41:17.566 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-11T14:41:17.566 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-11T14:41:17.566 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_dump_cache_on_map": "false", 2024-09-11T14:41:17.566 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-11T14:41:17.566 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-11T14:41:17.566 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_early_reply": "true", 2024-09-11T14:41:17.566 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_enable_op_tracker": "true", 2024-09-11T14:41:17.567 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_enforce_unique_name": "true", 2024-09-11T14:41:17.567 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-11T14:41:17.567 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-11T14:41:17.567 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_export_ephemeral_random": "true", 2024-09-11T14:41:17.567 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-11T14:41:17.567 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-11T14:41:17.567 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-11T14:41:17.567 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-11T14:41:17.567 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-11T14:41:17.567 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-11T14:41:17.567 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-11T14:41:17.567 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-11T14:41:17.567 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_health_summarize_threshold": "10", 2024-09-11T14:41:17.567 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-11T14:41:17.568 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-11T14:41:17.568 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_inject_health_dummy": "false", 2024-09-11T14:41:17.568 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-11T14:41:17.568 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_inject_migrator_session_race": "false", 2024-09-11T14:41:17.568 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-11T14:41:17.568 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-11T14:41:17.568 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-11T14:41:17.568 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_join_fs": "", 2024-09-11T14:41:17.568 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_journal_format": "1", 2024-09-11T14:41:17.568 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-11T14:41:17.568 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_kill_create_at": "0", 2024-09-11T14:41:17.568 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_kill_dirfrag_at": "0", 2024-09-11T14:41:17.569 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_kill_export_at": "0", 2024-09-11T14:41:17.569 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_kill_import_at": "0", 2024-09-11T14:41:17.569 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_kill_journal_at": "0", 2024-09-11T14:41:17.569 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_kill_journal_expire_at": "0", 2024-09-11T14:41:17.569 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_kill_journal_replay_at": "0", 2024-09-11T14:41:17.569 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_kill_link_at": "0", 2024-09-11T14:41:17.569 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_kill_mdstable_at": "0", 2024-09-11T14:41:17.569 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_kill_openc_at": "0", 2024-09-11T14:41:17.569 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_kill_rename_at": "0", 2024-09-11T14:41:17.569 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_kill_shutdown_at": "0", 2024-09-11T14:41:17.569 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_log_event_large_threshold": "524288", 2024-09-11T14:41:17.569 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_log_events_per_segment": "1024", 2024-09-11T14:41:17.569 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-11T14:41:17.570 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_log_max_events": "-1", 2024-09-11T14:41:17.570 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_log_max_segments": "128", 2024-09-11T14:41:17.570 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_log_pause": "false", 2024-09-11T14:41:17.570 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_log_segment_size": "0", 2024-09-11T14:41:17.570 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-11T14:41:17.570 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-11T14:41:17.570 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-11T14:41:17.570 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_log_trim_threshold": "128", 2024-09-11T14:41:17.570 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-11T14:41:17.570 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_log_warn_factor": "2.000000", 2024-09-11T14:41:17.570 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_max_caps_per_client": "1048576", 2024-09-11T14:41:17.570 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_max_completed_flushes": "100000", 2024-09-11T14:41:17.570 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_max_completed_requests": "100000", 2024-09-11T14:41:17.571 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_max_export_size": "20971520", 2024-09-11T14:41:17.571 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_max_file_recover": "32", 2024-09-11T14:41:17.571 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_max_purge_files": "64", 2024-09-11T14:41:17.571 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_max_purge_ops": "8192", 2024-09-11T14:41:17.571 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-11T14:41:17.571 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-11T14:41:17.571 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_max_snaps_per_dir": "100", 2024-09-11T14:41:17.571 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_metrics_update_interval": "2", 2024-09-11T14:41:17.571 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_min_caps_per_client": "100", 2024-09-11T14:41:17.571 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_min_caps_working_set": "10000", 2024-09-11T14:41:17.571 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-11T14:41:17.571 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_numa_node": "-1", 2024-09-11T14:41:17.571 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-11T14:41:17.571 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_op_complaint_time": "30.000000", 2024-09-11T14:41:17.572 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_op_history_duration": "600", 2024-09-11T14:41:17.572 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_op_history_size": "20", 2024-09-11T14:41:17.572 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_op_history_slow_op_size": "20", 2024-09-11T14:41:17.572 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-11T14:41:17.572 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_op_log_threshold": "5", 2024-09-11T14:41:17.572 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_ping_grace": "15", 2024-09-11T14:41:17.572 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_ping_interval": "5", 2024-09-11T14:41:17.572 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-11T14:41:17.572 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-11T14:41:17.572 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_recall_max_caps": "30000", 2024-09-11T14:41:17.572 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-11T14:41:17.572 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-11T14:41:17.572 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-11T14:41:17.573 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_recall_warning_threshold": "262144", 2024-09-11T14:41:17.573 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-11T14:41:17.573 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_replay_interval": "1.000000", 2024-09-11T14:41:17.573 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-11T14:41:17.573 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-11T14:41:17.573 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_root_ino_gid": "0", 2024-09-11T14:41:17.573 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_root_ino_uid": "0", 2024-09-11T14:41:17.573 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-11T14:41:17.573 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_scrub_stats_review_period": "1", 2024-09-11T14:41:17.573 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-09-11T14:41:17.573 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-09-11T14:41:17.573 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-11T14:41:17.573 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-11T14:41:17.574 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-11T14:41:17.574 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-11T14:41:17.574 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-11T14:41:17.574 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-11T14:41:17.574 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-11T14:41:17.574 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-11T14:41:17.574 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-11T14:41:17.574 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_shutdown_check": "0", 2024-09-11T14:41:17.574 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_skip_ino": "0", 2024-09-11T14:41:17.574 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-11T14:41:17.574 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_snap_max_uid": "4294967294", 2024-09-11T14:41:17.574 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_snap_min_uid": "0", 2024-09-11T14:41:17.574 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_snap_rstat": "false", 2024-09-11T14:41:17.575 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_standby_replay_damaged": "false", 2024-09-11T14:41:17.575 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_symlink_recovery": "true", 2024-09-11T14:41:17.575 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-11T14:41:17.575 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_thrash_exports": "0", 2024-09-11T14:41:17.575 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_thrash_fragments": "0", 2024-09-11T14:41:17.575 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_tick_interval": "5.000000", 2024-09-11T14:41:17.575 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_valgrind_exit": "false", 2024-09-11T14:41:17.575 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_verify_backtrace": "1", 2024-09-11T14:41:17.575 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_verify_scatter": "false", 2024-09-11T14:41:17.575 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-11T14:41:17.575 INFO:teuthology.orchestra.run.smithi181.stdout: "mds_wipe_sessions": "false", 2024-09-11T14:41:17.575 INFO:teuthology.orchestra.run.smithi181.stdout: "mempool_debug": "false", 2024-09-11T14:41:17.575 INFO:teuthology.orchestra.run.smithi181.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-11T14:41:17.575 INFO:teuthology.orchestra.run.smithi181.stdout: "memstore_device_bytes": "1073741824", 2024-09-11T14:41:17.576 INFO:teuthology.orchestra.run.smithi181.stdout: "memstore_page_set": "false", 2024-09-11T14:41:17.576 INFO:teuthology.orchestra.run.smithi181.stdout: "memstore_page_size": "65536", 2024-09-11T14:41:17.576 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_client_bytes": "134217728", 2024-09-11T14:41:17.576 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_client_messages": "512", 2024-09-11T14:41:17.576 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-11T14:41:17.576 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-11T14:41:17.576 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-11T14:41:17.576 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-11T14:41:17.576 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_disabled_modules": "", 2024-09-11T14:41:17.576 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_enable_op_tracker": "true", 2024-09-11T14:41:17.576 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-11T14:41:17.576 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_max_pg_creating": "1024", 2024-09-11T14:41:17.577 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_max_pg_num_change": "128", 2024-09-11T14:41:17.577 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_mds_bytes": "134217728", 2024-09-11T14:41:17.577 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_mds_messages": "128", 2024-09-11T14:41:17.577 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-11T14:41:17.577 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_mon_bytes": "134217728", 2024-09-11T14:41:17.577 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_mon_messages": "128", 2024-09-11T14:41:17.577 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-11T14:41:17.577 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-11T14:41:17.577 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_op_history_duration": "600", 2024-09-11T14:41:17.577 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_op_history_size": "20", 2024-09-11T14:41:17.577 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-11T14:41:17.577 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-11T14:41:17.577 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_op_log_threshold": "5", 2024-09-11T14:41:17.577 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_osd_bytes": "536870912", 2024-09-11T14:41:17.578 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_osd_messages": "8192", 2024-09-11T14:41:17.578 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_pool": "true", 2024-09-11T14:41:17.578 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-11T14:41:17.578 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_standby_modules": "true", 2024-09-11T14:41:17.578 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_stats_period": "5", 2024-09-11T14:41:17.578 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_stats_threshold": "5", 2024-09-11T14:41:17.578 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_tick_period": "2", 2024-09-11T14:41:17.578 INFO:teuthology.orchestra.run.smithi181.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-11T14:41:17.578 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-11T14:41:17.578 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_allow_pool_delete": "true", 2024-09-11T14:41:17.578 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_allow_pool_size_one": "true", 2024-09-11T14:41:17.578 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_auth_validate_all_caps": "true", 2024-09-11T14:41:17.578 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-11T14:41:17.579 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-11T14:41:17.579 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_client_bytes": "104857600", 2024-09-11T14:41:17.579 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_client_directed_command_retry": "2", 2024-09-11T14:41:17.579 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-11T14:41:17.579 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-11T14:41:17.579 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-11T14:41:17.579 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-11T14:41:17.579 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_client_hunt_parallel": "3", 2024-09-11T14:41:17.579 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_client_log_interval": "1.000000", 2024-09-11T14:41:17.580 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-11T14:41:17.580 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_client_ping_interval": "10.000000", 2024-09-11T14:41:17.580 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-11T14:41:17.580 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_client_target_rank": "-1", 2024-09-11T14:41:17.580 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-11T14:41:17.580 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-11T14:41:17.580 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-11T14:41:17.580 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_cluster_log_level": "debug", 2024-09-11T14:41:17.580 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_cluster_log_to_file": "true", 2024-09-11T14:41:17.580 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-11T14:41:17.580 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-11T14:41:17.580 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-11T14:41:17.580 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_cluster_log_to_journald": "false", 2024-09-11T14:41:17.581 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-11T14:41:17.581 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-11T14:41:17.581 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-11T14:41:17.581 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_compact_on_bootstrap": "false", 2024-09-11T14:41:17.581 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_compact_on_start": "false", 2024-09-11T14:41:17.581 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_compact_on_trim": "true", 2024-09-11T14:41:17.581 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-11T14:41:17.581 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-11T14:41:17.581 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-11T14:41:17.581 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_cpu_threads": "4", 2024-09-11T14:41:17.581 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_crush_min_required_version": "hammer", 2024-09-11T14:41:17.581 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_daemon_bytes": "419430400", 2024-09-11T14:41:17.581 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-11T14:41:17.582 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_data_avail_crit": "5", 2024-09-11T14:41:17.582 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_data_avail_warn": "5", 2024-09-11T14:41:17.582 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_data_size_warn": "16106127360", 2024-09-11T14:41:17.582 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-11T14:41:17.582 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-11T14:41:17.582 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_debug_dump_json": "false", 2024-09-11T14:41:17.582 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-11T14:41:17.582 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_debug_dump_transactions": "false", 2024-09-11T14:41:17.582 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_debug_extra_checks": "false", 2024-09-11T14:41:17.582 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-11T14:41:17.582 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-11T14:41:17.582 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_debug_no_require_reef": "false", 2024-09-11T14:41:17.582 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_debug_no_require_squid": "false", 2024-09-11T14:41:17.582 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-11T14:41:17.583 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-11T14:41:17.583 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-11T14:41:17.583 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_down_mkfs_grace": "120", 2024-09-11T14:41:17.583 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_down_uptime_grace": "60", 2024-09-11T14:41:17.583 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_election_default_strategy": "1", 2024-09-11T14:41:17.583 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_election_timeout": "5.000000", 2024-09-11T14:41:17.583 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-11T14:41:17.583 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_elector_ping_divisor": "2", 2024-09-11T14:41:17.583 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-11T14:41:17.583 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_enable_op_tracker": "true", 2024-09-11T14:41:17.583 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_fake_pool_delete": "false", 2024-09-11T14:41:17.583 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_force_quorum_join": "false", 2024-09-11T14:41:17.583 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-11T14:41:17.584 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_globalid_prealloc": "10000", 2024-09-11T14:41:17.584 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_health_detail_to_clog": "false", 2024-09-11T14:41:17.584 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_health_log_update_period": "5", 2024-09-11T14:41:17.584 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_health_max_detail": "50", 2024-09-11T14:41:17.584 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_health_to_clog": "true", 2024-09-11T14:41:17.584 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_health_to_clog_interval": "600", 2024-09-11T14:41:17.584 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-11T14:41:17.584 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_host": "172.21.15.181", 2024-09-11T14:41:17.584 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_host_override": "", 2024-09-11T14:41:17.584 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_initial_members": "", 2024-09-11T14:41:17.584 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-11T14:41:17.584 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-11T14:41:17.584 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-11T14:41:17.585 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-11T14:41:17.585 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-11T14:41:17.585 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_lease": "5.000000", 2024-09-11T14:41:17.585 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-11T14:41:17.585 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-11T14:41:17.585 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_log_full_interval": "50", 2024-09-11T14:41:17.585 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_log_max": "10000", 2024-09-11T14:41:17.585 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_log_max_summary": "50", 2024-09-11T14:41:17.585 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-11T14:41:17.585 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_max_log_epochs": "500", 2024-09-11T14:41:17.585 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-11T14:41:17.585 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-11T14:41:17.585 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_max_nvmeof_epochs": "500", 2024-09-11T14:41:17.585 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_max_osd": "10000", 2024-09-11T14:41:17.586 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_max_pg_per_osd": "10000", 2024-09-11T14:41:17.586 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_max_pool_pg_num": "65536", 2024-09-11T14:41:17.586 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-11T14:41:17.586 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-11T14:41:17.586 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_mds_force_trim_to": "0", 2024-09-11T14:41:17.586 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_mds_skip_sanity": "false", 2024-09-11T14:41:17.586 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_memory_autotune": "true", 2024-09-11T14:41:17.586 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_memory_target": "2147483648", 2024-09-11T14:41:17.586 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_mgr_beacon_grace": "30", 2024-09-11T14:41:17.586 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-11T14:41:17.586 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_mgr_digest_period": "5", 2024-09-11T14:41:17.586 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_mgr_inactive_grace": "60", 2024-09-11T14:41:17.586 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-11T14:41:17.587 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-11T14:41:17.587 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_min_osdmap_epochs": "50", 2024-09-11T14:41:17.587 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-09-11T14:41:17.587 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-09-11T14:41:17.587 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_op_complaint_time": "30", 2024-09-11T14:41:17.587 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_op_history_duration": "600", 2024-09-11T14:41:17.587 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_op_history_size": "20", 2024-09-11T14:41:17.587 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_op_history_slow_op_size": "20", 2024-09-11T14:41:17.587 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-11T14:41:17.587 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_op_log_threshold": "5", 2024-09-11T14:41:17.587 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-11T14:41:17.587 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-11T14:41:17.587 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-11T14:41:17.587 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_auto_mark_in": "false", 2024-09-11T14:41:17.588 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-11T14:41:17.588 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-11T14:41:17.588 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-11T14:41:17.588 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_cache_size": "500", 2024-09-11T14:41:17.588 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-11T14:41:17.588 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-11T14:41:17.588 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-11T14:41:17.588 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_down_out_interval": "600", 2024-09-11T14:41:17.588 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-11T14:41:17.588 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-11T14:41:17.588 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_force_trim_to": "0", 2024-09-11T14:41:17.588 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-11T14:41:17.588 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-11T14:41:17.589 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-11T14:41:17.589 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-11T14:41:17.589 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-11T14:41:17.589 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-11T14:41:17.589 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-11T14:41:17.589 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_min_down_reporters": "2", 2024-09-11T14:41:17.589 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-11T14:41:17.589 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-11T14:41:17.589 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-11T14:41:17.589 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-11T14:41:17.589 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-11T14:41:17.589 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-11T14:41:17.589 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_report_timeout": "900", 2024-09-11T14:41:17.589 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-11T14:41:17.590 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-11T14:41:17.590 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-11T14:41:17.590 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-11T14:41:17.590 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-11T14:41:17.590 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osdmap_full_prune_interval": "2", 2024-09-11T14:41:17.590 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osdmap_full_prune_min": "15", 2024-09-11T14:41:17.590 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-09-11T14:41:17.590 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-11T14:41:17.590 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_pg_stuck_threshold": "60", 2024-09-11T14:41:17.590 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-11T14:41:17.590 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-11T14:41:17.590 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-11T14:41:17.590 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-11T14:41:17.591 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-11T14:41:17.591 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-11T14:41:17.591 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_probe_timeout": "2.000000", 2024-09-11T14:41:17.591 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_reweight_max_change": "0.050000", 2024-09-11T14:41:17.591 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_reweight_max_osds": "4", 2024-09-11T14:41:17.591 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-11T14:41:17.591 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-11T14:41:17.591 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-11T14:41:17.591 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-11T14:41:17.591 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-11T14:41:17.591 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_scrub_interval": "86400", 2024-09-11T14:41:17.591 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_scrub_max_keys": "100", 2024-09-11T14:41:17.591 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_scrub_timeout": "300", 2024-09-11T14:41:17.592 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_session_timeout": "300", 2024-09-11T14:41:17.592 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_smart_report_timeout": "5", 2024-09-11T14:41:17.592 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_stat_smooth_intervals": "6", 2024-09-11T14:41:17.592 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-11T14:41:17.592 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_stretch_pool_min_size": "2", 2024-09-11T14:41:17.592 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_stretch_pool_size": "4", 2024-09-11T14:41:17.592 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-11T14:41:17.592 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-11T14:41:17.592 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_sync_debug": "false", 2024-09-11T14:41:17.592 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-11T14:41:17.592 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-11T14:41:17.592 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_sync_provider_kill_at": "0", 2024-09-11T14:41:17.592 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_sync_requester_kill_at": "0", 2024-09-11T14:41:17.593 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_sync_timeout": "60.000000", 2024-09-11T14:41:17.593 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_target_pg_per_osd": "100", 2024-09-11T14:41:17.593 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_tick_interval": "5", 2024-09-11T14:41:17.593 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_timecheck_interval": "300.000000", 2024-09-11T14:41:17.593 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-11T14:41:17.593 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_use_min_delay_socket": "false", 2024-09-11T14:41:17.593 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_warn_older_version_delay": "604800", 2024-09-11T14:41:17.593 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-11T14:41:17.593 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-11T14:41:17.593 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-11T14:41:17.593 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_warn_on_filestore_osds": "false", 2024-09-11T14:41:17.593 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-11T14:41:17.593 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-11T14:41:17.594 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-11T14:41:17.594 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_warn_on_misplaced": "false", 2024-09-11T14:41:17.594 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-11T14:41:17.594 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_warn_on_older_version": "true", 2024-09-11T14:41:17.594 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-11T14:41:17.594 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-11T14:41:17.594 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-09-11T14:41:17.594 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-11T14:41:17.594 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-11T14:41:17.594 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-11T14:41:17.594 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-11T14:41:17.594 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-11T14:41:17.594 INFO:teuthology.orchestra.run.smithi181.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-11T14:41:17.595 INFO:teuthology.orchestra.run.smithi181.stdout: "monmap": "", 2024-09-11T14:41:17.595 INFO:teuthology.orchestra.run.smithi181.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-11T14:41:17.595 INFO:teuthology.orchestra.run.smithi181.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-11T14:41:17.595 INFO:teuthology.orchestra.run.smithi181.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-11T14:41:17.595 INFO:teuthology.orchestra.run.smithi181.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-11T14:41:17.595 INFO:teuthology.orchestra.run.smithi181.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-11T14:41:17.595 INFO:teuthology.orchestra.run.smithi181.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-11T14:41:17.595 INFO:teuthology.orchestra.run.smithi181.stdout: "motr_tracing_enabled": "false", 2024-09-11T14:41:17.595 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_async_op_threads": "3", 2024-09-11T14:41:17.595 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-11T14:41:17.595 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_async_rdma_cm": "false", 2024-09-11T14:41:17.595 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_async_rdma_device_name": "", 2024-09-11T14:41:17.595 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_async_rdma_dscp": "96", 2024-09-11T14:41:17.595 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-11T14:41:17.596 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-11T14:41:17.596 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_async_rdma_local_gid": "", 2024-09-11T14:41:17.596 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-11T14:41:17.596 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_async_rdma_port_num": "1", 2024-09-11T14:41:17.596 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-11T14:41:17.596 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-11T14:41:17.596 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-11T14:41:17.596 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-11T14:41:17.596 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_async_rdma_sl": "3", 2024-09-11T14:41:17.596 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_async_rdma_support_srq": "true", 2024-09-11T14:41:17.596 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_async_rdma_type": "ib", 2024-09-11T14:41:17.596 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_async_reap_threshold": "5", 2024-09-11T14:41:17.596 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_bind_before_connect": "false", 2024-09-11T14:41:17.596 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_bind_ipv4": "true", 2024-09-11T14:41:17.597 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_bind_ipv6": "false", 2024-09-11T14:41:17.597 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_bind_msgr1": "true", 2024-09-11T14:41:17.597 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_bind_msgr2": "true", 2024-09-11T14:41:17.597 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_bind_port_max": "7568", 2024-09-11T14:41:17.597 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_bind_port_min": "6800", 2024-09-11T14:41:17.597 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-11T14:41:17.597 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_bind_retry_count": "3", 2024-09-11T14:41:17.597 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_bind_retry_delay": "5", 2024-09-11T14:41:17.597 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_blackhole_client": "false", 2024-09-11T14:41:17.597 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_blackhole_mds": "false", 2024-09-11T14:41:17.597 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_blackhole_mgr": "false", 2024-09-11T14:41:17.597 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_blackhole_mon": "false", 2024-09-11T14:41:17.597 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_blackhole_osd": "false", 2024-09-11T14:41:17.597 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_client_mode": "crc", 2024-09-11T14:41:17.598 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-09-11T14:41:17.598 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_cluster_mode": "crc", 2024-09-11T14:41:17.598 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_cluster_type": "", 2024-09-11T14:41:17.598 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_compress_secure": "false", 2024-09-11T14:41:17.598 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_connection_idle_timeout": "900", 2024-09-11T14:41:17.598 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_connection_ready_timeout": "10", 2024-09-11T14:41:17.598 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_crc_data": "true", 2024-09-11T14:41:17.598 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_crc_header": "true", 2024-09-11T14:41:17.598 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_die_on_bad_msg": "false", 2024-09-11T14:41:17.598 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_die_on_bug": "true", 2024-09-11T14:41:17.598 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_die_on_old_message": "true", 2024-09-11T14:41:17.598 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_die_on_skipped_message": "false", 2024-09-11T14:41:17.598 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-11T14:41:17.599 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-11T14:41:17.599 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_dpdk_coremask": "0xF", 2024-09-11T14:41:17.599 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-11T14:41:17.599 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-11T14:41:17.599 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_dpdk_enable_tso": "true", 2024-09-11T14:41:17.599 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-11T14:41:17.599 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-11T14:41:17.599 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_dpdk_hugepages": "", 2024-09-11T14:41:17.599 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-11T14:41:17.599 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-11T14:41:17.599 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_dpdk_lro": "true", 2024-09-11T14:41:17.599 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_dpdk_memory_channel": "4", 2024-09-11T14:41:17.600 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-11T14:41:17.600 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_dpdk_pmd": "", 2024-09-11T14:41:17.600 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_dpdk_port_id": "0", 2024-09-11T14:41:17.600 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-11T14:41:17.600 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-11T14:41:17.600 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_dump_on_send": "false", 2024-09-11T14:41:17.600 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_initial_backoff": "0.200000", 2024-09-11T14:41:17.600 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_inject_delay_max": "1.000000", 2024-09-11T14:41:17.600 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-11T14:41:17.600 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_inject_delay_type": "", 2024-09-11T14:41:17.600 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-11T14:41:17.600 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_inject_network_congestion": "0", 2024-09-11T14:41:17.600 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_inject_socket_failures": "0", 2024-09-11T14:41:17.600 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_learn_addr_from_peer": "true", 2024-09-11T14:41:17.601 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_max_accept_failures": "4", 2024-09-11T14:41:17.601 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_max_backoff": "15.000000", 2024-09-11T14:41:17.601 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_mon_client_mode": "crc", 2024-09-11T14:41:17.601 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_mon_cluster_mode": "crc", 2024-09-11T14:41:17.601 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_mon_service_mode": "crc", 2024-09-11T14:41:17.601 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_osd_compress_min_size": "1024", 2024-09-11T14:41:17.601 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_osd_compress_mode": "none", 2024-09-11T14:41:17.601 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-11T14:41:17.601 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-11T14:41:17.601 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_pq_min_cost": "65536", 2024-09-11T14:41:17.601 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_public_type": "", 2024-09-11T14:41:17.601 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_service_mode": "crc", 2024-09-11T14:41:17.601 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_tcp_listen_backlog": "512", 2024-09-11T14:41:17.602 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_tcp_nodelay": "true", 2024-09-11T14:41:17.602 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-11T14:41:17.602 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_tcp_rcvbuf": "0", 2024-09-11T14:41:17.602 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_time_events_min_wait_interval": "1000", 2024-09-11T14:41:17.602 INFO:teuthology.orchestra.run.smithi181.stdout: "ms_type": "async+posix", 2024-09-11T14:41:17.602 INFO:teuthology.orchestra.run.smithi181.stdout: "no_config_file": "false", 2024-09-11T14:41:17.602 INFO:teuthology.orchestra.run.smithi181.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-09-11T14:41:17.602 INFO:teuthology.orchestra.run.smithi181.stdout: "nvmeof_mon_client_tick_period": "2", 2024-09-11T14:41:17.602 INFO:teuthology.orchestra.run.smithi181.stdout: "objecter_completion_locks_per_session": "32", 2024-09-11T14:41:17.602 INFO:teuthology.orchestra.run.smithi181.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-11T14:41:17.602 INFO:teuthology.orchestra.run.smithi181.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-11T14:41:17.602 INFO:teuthology.orchestra.run.smithi181.stdout: "objecter_inflight_ops": "1024", 2024-09-11T14:41:17.602 INFO:teuthology.orchestra.run.smithi181.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-11T14:41:17.602 INFO:teuthology.orchestra.run.smithi181.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-11T14:41:17.603 INFO:teuthology.orchestra.run.smithi181.stdout: "objecter_tick_interval": "5.000000", 2024-09-11T14:41:17.603 INFO:teuthology.orchestra.run.smithi181.stdout: "objecter_timeout": "10.000000", 2024-09-11T14:41:17.603 INFO:teuthology.orchestra.run.smithi181.stdout: "objectstore_blackhole": "false", 2024-09-11T14:41:17.603 INFO:teuthology.orchestra.run.smithi181.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-11T14:41:17.603 INFO:teuthology.orchestra.run.smithi181.stdout: "openssl_engine_opts": "", 2024-09-11T14:41:17.603 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_agent_delay_time": "5.000000", 2024-09-11T14:41:17.603 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_agent_hist_halflife": "1000", 2024-09-11T14:41:17.603 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_agent_max_low_ops": "2", 2024-09-11T14:41:17.603 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_agent_max_ops": "4", 2024-09-11T14:41:17.603 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-11T14:41:17.603 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-11T14:41:17.603 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_agent_slop": "0.020000", 2024-09-11T14:41:17.603 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-11T14:41:17.603 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-11T14:41:17.604 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_asio_thread_count": "2", 2024-09-11T14:41:17.604 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_async_recovery_min_cost": "100", 2024-09-11T14:41:17.604 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-11T14:41:17.604 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-11T14:41:17.604 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_backfill_scan_max": "512", 2024-09-11T14:41:17.604 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_backfill_scan_min": "64", 2024-09-11T14:41:17.604 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_backoff_on_degraded": "false", 2024-09-11T14:41:17.604 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_backoff_on_peering": "false", 2024-09-11T14:41:17.604 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_backoff_on_unfound": "true", 2024-09-11T14:41:17.604 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_beacon_report_interval": "300", 2024-09-11T14:41:17.604 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_bench_duration": "30", 2024-09-11T14:41:17.604 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-11T14:41:17.604 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_bench_max_block_size": "67108864", 2024-09-11T14:41:17.604 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-11T14:41:17.605 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_blkin_trace_all": "false", 2024-09-11T14:41:17.605 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-11T14:41:17.605 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-11T14:41:17.605 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-11T14:41:17.605 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-11T14:41:17.605 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_check_for_log_corruption": "false", 2024-09-11T14:41:17.605 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-11T14:41:17.605 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-11T14:41:17.605 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-11T14:41:17.605 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-11T14:41:17.605 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_class_update_on_start": "true", 2024-09-11T14:41:17.605 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_client_message_cap": "256", 2024-09-11T14:41:17.605 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_client_message_size_cap": "524288000", 2024-09-11T14:41:17.606 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_client_op_priority": "63", 2024-09-11T14:41:17.606 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_client_watch_timeout": "30", 2024-09-11T14:41:17.606 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_command_max_records": "256", 2024-09-11T14:41:17.606 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_compact_on_start": "false", 2024-09-11T14:41:17.606 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-11T14:41:17.606 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_crush_chooseleaf_type": "0", 2024-09-11T14:41:17.606 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-11T14:41:17.606 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_crush_update_on_start": "true", 2024-09-11T14:41:17.606 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_crush_update_weight_set": "true", 2024-09-11T14:41:17.606 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-11T14:41:17.606 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-11T14:41:17.606 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-11T14:41:17.606 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-11T14:41:17.607 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-11T14:41:17.607 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_debug_feed_pullee": "-1", 2024-09-11T14:41:17.607 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-11T14:41:17.607 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-11T14:41:17.607 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-11T14:41:17.607 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_debug_misdirected_ops": "false", 2024-09-11T14:41:17.607 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_debug_no_acting_change": "false", 2024-09-11T14:41:17.607 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_debug_no_purge_strays": "false", 2024-09-11T14:41:17.607 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_debug_op_order": "false", 2024-09-11T14:41:17.607 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-11T14:41:17.607 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-11T14:41:17.607 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-11T14:41:17.607 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-11T14:41:17.608 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_debug_shutdown": "false", 2024-09-11T14:41:17.608 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-11T14:41:17.608 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-11T14:41:17.608 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_debug_trim_objects": "false", 2024-09-11T14:41:17.608 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-11T14:41:17.608 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-11T14:41:17.608 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_debug_verify_snaps": "false", 2024-09-11T14:41:17.608 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-11T14:41:17.608 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-11T14:41:17.608 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_deep_scrub_keys": "1024", 2024-09-11T14:41:17.608 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-11T14:41:17.608 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-11T14:41:17.609 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-11T14:41:17.609 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_deep_scrub_stride": "524288", 2024-09-11T14:41:17.609 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-11T14:41:17.609 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_default_data_pool_replay_window": "5", 2024-09-11T14:41:17.609 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_default_notify_timeout": "30", 2024-09-11T14:41:17.609 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_delete_sleep": "0.000000", 2024-09-11T14:41:17.609 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-11T14:41:17.609 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-11T14:41:17.609 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-11T14:41:17.609 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_discard_disconnected_ops": "true", 2024-09-11T14:41:17.609 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_ec_partial_reads": "true", 2024-09-11T14:41:17.609 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_enable_op_tracker": "true", 2024-09-11T14:41:17.609 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-11T14:41:17.609 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-11T14:41:17.610 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-11T14:41:17.610 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-11T14:41:17.610 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_fast_info": "true", 2024-09-11T14:41:17.610 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_fast_shutdown": "true", 2024-09-11T14:41:17.610 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-11T14:41:17.610 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-11T14:41:17.610 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-11T14:41:17.610 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-11T14:41:17.610 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-11T14:41:17.610 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_function_tracing": "false", 2024-09-11T14:41:17.610 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_heartbeat_grace": "20", 2024-09-11T14:41:17.610 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_heartbeat_interval": "6", 2024-09-11T14:41:17.610 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-11T14:41:17.611 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_heartbeat_min_peers": "10", 2024-09-11T14:41:17.611 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_heartbeat_min_size": "2000", 2024-09-11T14:41:17.611 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_heartbeat_stale": "600", 2024-09-11T14:41:17.611 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-11T14:41:17.611 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_hit_set_max_size": "100000", 2024-09-11T14:41:17.611 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_hit_set_min_size": "1000", 2024-09-11T14:41:17.611 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-11T14:41:17.611 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-11T14:41:17.611 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-11T14:41:17.611 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-11T14:41:17.611 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-11T14:41:17.611 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-11T14:41:17.611 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_journal_size": "5120", 2024-09-11T14:41:17.611 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_kill_backfill_at": "0", 2024-09-11T14:41:17.612 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-11T14:41:17.612 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_map_cache_size": "50", 2024-09-11T14:41:17.612 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_map_dedup": "true", 2024-09-11T14:41:17.612 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_map_message_max": "40", 2024-09-11T14:41:17.612 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-11T14:41:17.612 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_map_share_max_epochs": "40", 2024-09-11T14:41:17.612 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_max_attr_name_len": "100", 2024-09-11T14:41:17.612 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_max_attr_size": "0", 2024-09-11T14:41:17.612 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_max_backfills": "1", 2024-09-11T14:41:17.612 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_max_markdown_count": "5", 2024-09-11T14:41:17.612 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_max_markdown_period": "600", 2024-09-11T14:41:17.612 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_max_object_name_len": "2048", 2024-09-11T14:41:17.612 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_max_object_namespace_len": "256", 2024-09-11T14:41:17.613 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_max_object_size": "134217728", 2024-09-11T14:41:17.613 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-11T14:41:17.613 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-11T14:41:17.613 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_max_pg_blocked_by": "16", 2024-09-11T14:41:17.613 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_max_pg_log_entries": "10000", 2024-09-11T14:41:17.613 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-11T14:41:17.613 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_max_pgls": "1024", 2024-09-11T14:41:17.613 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_max_push_cost": "8388608", 2024-09-11T14:41:17.613 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_max_push_objects": "10", 2024-09-11T14:41:17.613 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_max_scrubs": "3", 2024-09-11T14:41:17.613 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-11T14:41:17.613 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_max_trimming_pgs": "2", 2024-09-11T14:41:17.613 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_max_write_op_reply_len": "64", 2024-09-11T14:41:17.613 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_max_write_size": "90", 2024-09-11T14:41:17.614 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-11T14:41:17.614 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-11T14:41:17.614 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-11T14:41:17.614 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-11T14:41:17.614 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-11T14:41:17.614 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-11T14:41:17.614 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-11T14:41:17.614 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-11T14:41:17.614 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mclock_profile": "balanced", 2024-09-11T14:41:17.614 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-11T14:41:17.614 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-11T14:41:17.614 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-11T14:41:17.614 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-11T14:41:17.615 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-11T14:41:17.615 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-11T14:41:17.615 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-11T14:41:17.615 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-11T14:41:17.615 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-11T14:41:17.615 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-11T14:41:17.615 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-11T14:41:17.615 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_memory_base": "805306368", 2024-09-11T14:41:17.615 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_memory_cache_min": "134217728", 2024-09-11T14:41:17.615 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-11T14:41:17.615 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-11T14:41:17.615 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_memory_target": "4294967296", 2024-09-11T14:41:17.615 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_memory_target_autotune": "false", 2024-09-11T14:41:17.615 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-11T14:41:17.616 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_min_pg_log_entries": "250", 2024-09-11T14:41:17.616 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_min_recovery_priority": "0", 2024-09-11T14:41:17.616 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-11T14:41:17.616 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-11T14:41:17.616 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mon_report_interval": "5", 2024-09-11T14:41:17.616 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-11T14:41:17.616 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-11T14:41:17.616 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_num_cache_shards": "32", 2024-09-11T14:41:17.616 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_num_op_tracker_shard": "32", 2024-09-11T14:41:17.616 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_numa_auto_affinity": "true", 2024-09-11T14:41:17.616 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_numa_node": "-1", 2024-09-11T14:41:17.616 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_numa_prefer_iface": "true", 2024-09-11T14:41:17.616 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-11T14:41:17.617 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_objecter_finishers": "1", 2024-09-11T14:41:17.617 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_objectstore": "bluestore", 2024-09-11T14:41:17.617 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_objectstore_fuse": "false", 2024-09-11T14:41:17.617 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_objectstore_ideal_list_max": "64", 2024-09-11T14:41:17.617 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_objectstore_tracing": "false", 2024-09-11T14:41:17.617 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_op_complaint_time": "30.000000", 2024-09-11T14:41:17.617 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_op_history_duration": "600", 2024-09-11T14:41:17.617 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_op_history_size": "20", 2024-09-11T14:41:17.617 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_op_history_slow_op_size": "20", 2024-09-11T14:41:17.617 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-11T14:41:17.617 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_op_log_threshold": "5", 2024-09-11T14:41:17.617 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_op_num_shards": "0", 2024-09-11T14:41:17.617 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_op_num_shards_hdd": "5", 2024-09-11T14:41:17.618 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_op_num_shards_ssd": "8", 2024-09-11T14:41:17.618 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-11T14:41:17.618 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-11T14:41:17.618 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-11T14:41:17.618 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-11T14:41:17.618 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_op_pq_min_cost": "65536", 2024-09-11T14:41:17.618 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-11T14:41:17.618 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_op_queue_cut_off": "high", 2024-09-11T14:41:17.618 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-11T14:41:17.618 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_op_thread_timeout": "15", 2024-09-11T14:41:17.618 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_open_classes_on_start": "true", 2024-09-11T14:41:17.618 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_os_flags": "0", 2024-09-11T14:41:17.618 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_peering_op_priority": "255", 2024-09-11T14:41:17.618 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pg_delete_cost": "1048576", 2024-09-11T14:41:17.619 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pg_delete_priority": "5", 2024-09-11T14:41:17.619 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-11T14:41:17.619 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-11T14:41:17.619 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-11T14:41:17.619 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pg_log_trim_max": "10000", 2024-09-11T14:41:17.619 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pg_log_trim_min": "100", 2024-09-11T14:41:17.619 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-11T14:41:17.619 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-11T14:41:17.619 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-11T14:41:17.619 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-11T14:41:17.619 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-11T14:41:17.619 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-11T14:41:17.619 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-11T14:41:17.620 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-11T14:41:17.620 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-11T14:41:17.620 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-11T14:41:17.620 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pool_default_crimson": "true", 2024-09-11T14:41:17.620 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-11T14:41:17.620 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-11T14:41:17.620 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-11T14:41:17.620 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-11T14:41:17.620 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-11T14:41:17.620 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-11T14:41:17.620 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-11T14:41:17.620 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-11T14:41:17.620 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pool_default_flags": "0", 2024-09-11T14:41:17.620 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-11T14:41:17.621 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pool_default_min_size": "2", 2024-09-11T14:41:17.621 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-11T14:41:17.621 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pool_default_pg_num": "32", 2024-09-11T14:41:17.621 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pool_default_pgp_num": "0", 2024-09-11T14:41:17.621 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-11T14:41:17.621 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pool_default_read_ratio": "70", 2024-09-11T14:41:17.621 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pool_default_size": "3", 2024-09-11T14:41:17.621 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pool_default_type": "replicated", 2024-09-11T14:41:17.621 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-11T14:41:17.621 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-11T14:41:17.621 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_push_per_object_cost": "1000", 2024-09-11T14:41:17.621 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-11T14:41:17.621 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_recover_clone_overlap": "true", 2024-09-11T14:41:17.622 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-11T14:41:17.622 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_recovery_cost": "20971520", 2024-09-11T14:41:17.622 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-11T14:41:17.622 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_recovery_max_active": "0", 2024-09-11T14:41:17.622 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-11T14:41:17.622 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-11T14:41:17.622 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-11T14:41:17.622 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-11T14:41:17.622 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_recovery_max_single_start": "1", 2024-09-11T14:41:17.622 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_recovery_op_priority": "3", 2024-09-11T14:41:17.622 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-11T14:41:17.622 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_recovery_priority": "5", 2024-09-11T14:41:17.622 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-11T14:41:17.623 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_recovery_sleep": "0.000000", 2024-09-11T14:41:17.623 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-11T14:41:17.623 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-11T14:41:17.623 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-11T14:41:17.623 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_repair_during_recovery": "false", 2024-09-11T14:41:17.623 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_requested_scrub_priority": "120", 2024-09-11T14:41:17.623 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-11T14:41:17.623 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-11T14:41:17.623 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_scrub_auto_repair": "false", 2024-09-11T14:41:17.623 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-11T14:41:17.623 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-11T14:41:17.623 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_scrub_begin_hour": "0", 2024-09-11T14:41:17.623 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_scrub_begin_week_day": "0", 2024-09-11T14:41:17.624 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_scrub_chunk_max": "25", 2024-09-11T14:41:17.624 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_scrub_chunk_min": "5", 2024-09-11T14:41:17.624 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_scrub_cost": "52428800", 2024-09-11T14:41:17.624 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-11T14:41:17.624 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_scrub_during_recovery": "false", 2024-09-11T14:41:17.624 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_scrub_end_hour": "0", 2024-09-11T14:41:17.624 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_scrub_end_week_day": "0", 2024-09-11T14:41:17.624 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_scrub_event_cost": "4096", 2024-09-11T14:41:17.624 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-11T14:41:17.624 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-11T14:41:17.624 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_scrub_invalid_stats": "true", 2024-09-11T14:41:17.624 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-11T14:41:17.624 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-11T14:41:17.624 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_scrub_max_preemptions": "5", 2024-09-11T14:41:17.625 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-11T14:41:17.625 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_scrub_priority": "5", 2024-09-11T14:41:17.625 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_scrub_retry_after_noscrub": "60", 2024-09-11T14:41:17.625 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_scrub_retry_delay": "30", 2024-09-11T14:41:17.625 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_scrub_retry_new_interval": "10", 2024-09-11T14:41:17.625 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_scrub_retry_pg_state": "60", 2024-09-11T14:41:17.625 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_scrub_retry_trimming": "10", 2024-09-11T14:41:17.625 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_scrub_sleep": "0.000000", 2024-09-11T14:41:17.625 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-11T14:41:17.625 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-11T14:41:17.625 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-11T14:41:17.625 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-09-11T14:41:17.625 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_skip_data_digest": "false", 2024-09-11T14:41:17.626 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_smart_report_timeout": "5", 2024-09-11T14:41:17.626 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_snap_trim_cost": "1048576", 2024-09-11T14:41:17.626 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_snap_trim_priority": "5", 2024-09-11T14:41:17.626 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-11T14:41:17.626 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-11T14:41:17.626 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-11T14:41:17.626 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-11T14:41:17.626 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-11T14:41:17.626 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-11T14:41:17.626 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-11T14:41:17.626 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_target_transaction_size": "30", 2024-09-11T14:41:17.626 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-11T14:41:17.626 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-11T14:41:17.627 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-11T14:41:17.627 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-11T14:41:17.627 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-11T14:41:17.627 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-11T14:41:17.627 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-11T14:41:17.627 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-11T14:41:17.627 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-11T14:41:17.627 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-11T14:41:17.627 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_tracing": "false", 2024-09-11T14:41:17.627 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_use_stale_snap": "false", 2024-09-11T14:41:17.627 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-11T14:41:17.627 INFO:teuthology.orchestra.run.smithi181.stdout: "osdc_blkin_trace_all": "false", 2024-09-11T14:41:17.627 INFO:teuthology.orchestra.run.smithi181.stdout: "paxos_kill_at": "0", 2024-09-11T14:41:17.628 INFO:teuthology.orchestra.run.smithi181.stdout: "paxos_max_join_drift": "10", 2024-09-11T14:41:17.628 INFO:teuthology.orchestra.run.smithi181.stdout: "paxos_min": "500", 2024-09-11T14:41:17.628 INFO:teuthology.orchestra.run.smithi181.stdout: "paxos_min_wait": "0.050000", 2024-09-11T14:41:17.628 INFO:teuthology.orchestra.run.smithi181.stdout: "paxos_propose_interval": "1.000000", 2024-09-11T14:41:17.628 INFO:teuthology.orchestra.run.smithi181.stdout: "paxos_service_trim_max": "500", 2024-09-11T14:41:17.628 INFO:teuthology.orchestra.run.smithi181.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-11T14:41:17.628 INFO:teuthology.orchestra.run.smithi181.stdout: "paxos_service_trim_min": "10", 2024-09-11T14:41:17.628 INFO:teuthology.orchestra.run.smithi181.stdout: "paxos_stash_full_interval": "25", 2024-09-11T14:41:17.628 INFO:teuthology.orchestra.run.smithi181.stdout: "paxos_trim_max": "500", 2024-09-11T14:41:17.628 INFO:teuthology.orchestra.run.smithi181.stdout: "paxos_trim_min": "250", 2024-09-11T14:41:17.628 INFO:teuthology.orchestra.run.smithi181.stdout: "perf": "true", 2024-09-11T14:41:17.628 INFO:teuthology.orchestra.run.smithi181.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-09-11T14:41:17.628 INFO:teuthology.orchestra.run.smithi181.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-11T14:41:17.629 INFO:teuthology.orchestra.run.smithi181.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-11T14:41:17.629 INFO:teuthology.orchestra.run.smithi181.stdout: "public_addr": "-", 2024-09-11T14:41:17.629 INFO:teuthology.orchestra.run.smithi181.stdout: "public_addrv": "", 2024-09-11T14:41:17.629 INFO:teuthology.orchestra.run.smithi181.stdout: "public_bind_addr": "-", 2024-09-11T14:41:17.629 INFO:teuthology.orchestra.run.smithi181.stdout: "public_network": "", 2024-09-11T14:41:17.629 INFO:teuthology.orchestra.run.smithi181.stdout: "public_network_interface": "", 2024-09-11T14:41:17.629 INFO:teuthology.orchestra.run.smithi181.stdout: "qat_compressor_busy_polling": "false", 2024-09-11T14:41:17.629 INFO:teuthology.orchestra.run.smithi181.stdout: "qat_compressor_enabled": "false", 2024-09-11T14:41:17.629 INFO:teuthology.orchestra.run.smithi181.stdout: "qat_compressor_session_max_number": "256", 2024-09-11T14:41:17.629 INFO:teuthology.orchestra.run.smithi181.stdout: "rados_mon_op_timeout": "0", 2024-09-11T14:41:17.629 INFO:teuthology.orchestra.run.smithi181.stdout: "rados_osd_op_timeout": "0", 2024-09-11T14:41:17.630 INFO:teuthology.orchestra.run.smithi181.stdout: "rados_tracing": "false", 2024-09-11T14:41:17.630 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_atime_update_interval": "60", 2024-09-11T14:41:17.630 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-11T14:41:17.630 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_balance_parent_reads": "false", 2024-09-11T14:41:17.630 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_balance_snap_reads": "false", 2024-09-11T14:41:17.630 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_blkin_trace_all": "false", 2024-09-11T14:41:17.630 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-11T14:41:17.630 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-11T14:41:17.631 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_cache": "true", 2024-09-11T14:41:17.631 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-11T14:41:17.631 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-11T14:41:17.631 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-11T14:41:17.631 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-11T14:41:17.631 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_cache_policy": "writearound", 2024-09-11T14:41:17.631 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_cache_size": "33554432", 2024-09-11T14:41:17.631 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-11T14:41:17.631 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-11T14:41:17.631 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_clone_copy_on_read": "false", 2024-09-11T14:41:17.632 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_compression_hint": "none", 2024-09-11T14:41:17.632 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_concurrent_management_ops": "10", 2024-09-11T14:41:17.632 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-11T14:41:17.632 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_default_clone_format": "auto", 2024-09-11T14:41:17.632 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_default_data_pool": "", 2024-09-11T14:41:17.632 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_default_features": "61", 2024-09-11T14:41:17.632 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_default_format": "2", 2024-09-11T14:41:17.632 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_default_map_options": "", 2024-09-11T14:41:17.632 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_default_order": "22", 2024-09-11T14:41:17.633 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_default_pool": "rbd", 2024-09-11T14:41:17.633 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-11T14:41:17.633 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_default_stripe_count": "0", 2024-09-11T14:41:17.633 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_default_stripe_unit": "0", 2024-09-11T14:41:17.633 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-11T14:41:17.633 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-11T14:41:17.633 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-11T14:41:17.633 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_enable_alloc_hint": "true", 2024-09-11T14:41:17.633 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-11T14:41:17.633 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_io_scheduler": "simple", 2024-09-11T14:41:17.633 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-11T14:41:17.633 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-11T14:41:17.634 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-11T14:41:17.634 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-11T14:41:17.634 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-11T14:41:17.634 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-11T14:41:17.634 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-11T14:41:17.634 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-11T14:41:17.634 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-11T14:41:17.634 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_journal_order": "24", 2024-09-11T14:41:17.634 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_journal_pool": "", 2024-09-11T14:41:17.634 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_journal_splay_width": "4", 2024-09-11T14:41:17.634 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_localize_parent_reads": "false", 2024-09-11T14:41:17.634 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_localize_snap_reads": "false", 2024-09-11T14:41:17.634 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-11T14:41:17.634 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-11T14:41:17.635 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-11T14:41:17.635 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-11T14:41:17.635 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-11T14:41:17.635 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-11T14:41:17.635 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-11T14:41:17.635 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-11T14:41:17.635 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-11T14:41:17.635 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-11T14:41:17.635 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-11T14:41:17.635 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-11T14:41:17.635 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-11T14:41:17.635 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-11T14:41:17.635 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-11T14:41:17.636 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-11T14:41:17.636 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-11T14:41:17.636 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-11T14:41:17.636 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-11T14:41:17.636 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-11T14:41:17.636 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-11T14:41:17.636 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-11T14:41:17.636 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-11T14:41:17.636 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-11T14:41:17.636 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-11T14:41:17.636 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-11T14:41:17.636 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-11T14:41:17.637 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-11T14:41:17.637 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-11T14:41:17.637 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-11T14:41:17.637 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-11T14:41:17.637 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_mtime_update_interval": "60", 2024-09-11T14:41:17.637 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_non_blocking_aio": "true", 2024-09-11T14:41:17.637 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_op_thread_timeout": "60", 2024-09-11T14:41:17.637 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_op_threads": "1", 2024-09-11T14:41:17.637 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_parent_cache_enabled": "false", 2024-09-11T14:41:17.637 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-11T14:41:17.637 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-11T14:41:17.637 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-11T14:41:17.637 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_plugins": "", 2024-09-11T14:41:17.638 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_qos_bps_burst": "0", 2024-09-11T14:41:17.638 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-11T14:41:17.638 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_qos_bps_limit": "0", 2024-09-11T14:41:17.638 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_qos_exclude_ops": "0", 2024-09-11T14:41:17.638 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_qos_iops_burst": "0", 2024-09-11T14:41:17.638 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-11T14:41:17.638 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_qos_iops_limit": "0", 2024-09-11T14:41:17.638 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-11T14:41:17.638 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-11T14:41:17.638 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-11T14:41:17.638 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-11T14:41:17.638 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-11T14:41:17.639 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-11T14:41:17.639 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-11T14:41:17.639 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-11T14:41:17.639 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-11T14:41:17.639 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-11T14:41:17.639 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-11T14:41:17.639 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-11T14:41:17.639 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-11T14:41:17.639 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-11T14:41:17.639 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_read_from_replica_policy": "default", 2024-09-11T14:41:17.639 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-11T14:41:17.639 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-11T14:41:17.639 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-11T14:41:17.639 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-11T14:41:17.640 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_skip_partial_discard": "true", 2024-09-11T14:41:17.640 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-11T14:41:17.640 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_tracing": "false", 2024-09-11T14:41:17.640 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_validate_names": "true", 2024-09-11T14:41:17.640 INFO:teuthology.orchestra.run.smithi181.stdout: "rbd_validate_pool": "true", 2024-09-11T14:41:17.640 INFO:teuthology.orchestra.run.smithi181.stdout: "restapi_base_url": "", 2024-09-11T14:41:17.640 INFO:teuthology.orchestra.run.smithi181.stdout: "restapi_log_level": "", 2024-09-11T14:41:17.640 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-11T14:41:17.640 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-11T14:41:17.640 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_acl_grants_max_num": "100", 2024-09-11T14:41:17.640 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_admin_entry": "admin", 2024-09-11T14:41:17.640 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-11T14:41:17.640 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_asio_assert_yielding": "false", 2024-09-11T14:41:17.641 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_backend_store": "rados", 2024-09-11T14:41:17.641 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_barbican_url": "", 2024-09-11T14:41:17.641 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_beast_enable_async": "true", 2024-09-11T14:41:17.641 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_bucket_counters_cache": "false", 2024-09-11T14:41:17.641 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-11T14:41:17.641 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-11T14:41:17.641 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-11T14:41:17.641 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-11T14:41:17.641 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-11T14:41:17.641 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-11T14:41:17.641 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-11T14:41:17.641 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-11T14:41:17.641 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_cache_enabled": "true", 2024-09-11T14:41:17.642 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_cache_expiry_interval": "900", 2024-09-11T14:41:17.642 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_cache_lru_size": "25000", 2024-09-11T14:41:17.642 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_config_store": "rados", 2024-09-11T14:41:17.642 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_content_length_compat": "false", 2024-09-11T14:41:17.642 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_copy_obj_progress": "true", 2024-09-11T14:41:17.642 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-11T14:41:17.642 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_cors_rules_max_num": "100", 2024-09-11T14:41:17.642 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_cross_domain_policy": "", 2024-09-11T14:41:17.642 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-11T14:41:17.642 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_kmip_addr": "", 2024-09-11T14:41:17.642 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-11T14:41:17.642 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-11T14:41:17.642 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-11T14:41:17.642 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-11T14:41:17.643 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_kmip_password": "", 2024-09-11T14:41:17.643 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-11T14:41:17.643 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_kmip_username": "", 2024-09-11T14:41:17.643 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_require_ssl": "true", 2024-09-11T14:41:17.643 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-11T14:41:17.643 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-11T14:41:17.643 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-11T14:41:17.643 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-11T14:41:17.643 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-11T14:41:17.643 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-11T14:41:17.643 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-11T14:41:17.643 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-11T14:41:17.643 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-11T14:41:17.644 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-11T14:41:17.644 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-11T14:41:17.644 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-11T14:41:17.644 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-11T14:41:17.644 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-11T14:41:17.644 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-11T14:41:17.644 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_vault_addr": "", 2024-09-11T14:41:17.644 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_vault_auth": "token", 2024-09-11T14:41:17.644 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_vault_namespace": "", 2024-09-11T14:41:17.644 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_vault_prefix": "", 2024-09-11T14:41:17.644 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-11T14:41:17.644 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-11T14:41:17.644 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-11T14:41:17.645 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-11T14:41:17.645 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_vault_token_file": "", 2024-09-11T14:41:17.645 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-11T14:41:17.645 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_curl_buffersize": "524288", 2024-09-11T14:41:17.645 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-11T14:41:17.645 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_curl_low_speed_time": "300", 2024-09-11T14:41:17.645 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-11T14:41:17.645 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-11T14:41:17.645 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-11T14:41:17.645 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-11T14:41:17.645 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-11T14:41:17.645 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-11T14:41:17.645 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-11T14:41:17.646 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-11T14:41:17.646 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-11T14:41:17.646 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-11T14:41:17.646 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-09-11T14:41:17.646 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-09-11T14:41:17.646 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-09-11T14:41:17.646 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-09-11T14:41:17.646 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-09-11T14:41:17.646 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_d4n_l1_fadvise": "4", 2024-09-11T14:41:17.646 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-09-11T14:41:17.646 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-09-11T14:41:17.646 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-11T14:41:17.646 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_data_log_changes_size": "1000", 2024-09-11T14:41:17.647 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_data_log_num_shards": "128", 2024-09-11T14:41:17.647 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-11T14:41:17.647 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_data_log_window": "30", 2024-09-11T14:41:17.647 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-11T14:41:17.647 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-11T14:41:17.647 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-11T14:41:17.647 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-11T14:41:17.647 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-11T14:41:17.647 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-11T14:41:17.647 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-11T14:41:17.647 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-11T14:41:17.647 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-11T14:41:17.647 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-11T14:41:17.648 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-11T14:41:17.648 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-11T14:41:17.648 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-11T14:41:17.648 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_disable_s3select": "false", 2024-09-11T14:41:17.648 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-11T14:41:17.648 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-11T14:41:17.648 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-11T14:41:17.648 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-11T14:41:17.648 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-11T14:41:17.648 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-11T14:41:17.648 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-11T14:41:17.648 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-11T14:41:17.648 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-11T14:41:17.649 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-11T14:41:17.649 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-11T14:41:17.649 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-11T14:41:17.649 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_dns_name": "", 2024-09-11T14:41:17.649 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_dns_s3website_name": "", 2024-09-11T14:41:17.649 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_dynamic_resharding": "true", 2024-09-11T14:41:17.649 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-09-11T14:41:17.649 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-09-11T14:41:17.649 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-11T14:41:17.649 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_enable_gc_threads": "true", 2024-09-11T14:41:17.649 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_enable_lc_threads": "true", 2024-09-11T14:41:17.649 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_enable_mdsearch": "true", 2024-09-11T14:41:17.649 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_enable_ops_log": "false", 2024-09-11T14:41:17.650 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_enable_quota_threads": "true", 2024-09-11T14:41:17.650 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_enable_static_website": "false", 2024-09-11T14:41:17.650 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_enable_usage_log": "false", 2024-09-11T14:41:17.650 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_enforce_swift_acls": "true", 2024-09-11T14:41:17.650 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_exit_timeout_secs": "120", 2024-09-11T14:41:17.650 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_expose_bucket": "false", 2024-09-11T14:41:17.650 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_extended_http_attrs": "", 2024-09-11T14:41:17.650 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_filter": "none", 2024-09-11T14:41:17.650 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-11T14:41:17.650 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_frontends": "beast port=7480", 2024-09-11T14:41:17.650 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-11T14:41:17.650 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_gc_max_deferred": "50", 2024-09-11T14:41:17.650 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-11T14:41:17.650 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_gc_max_objs": "32", 2024-09-11T14:41:17.651 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-11T14:41:17.651 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-11T14:41:17.651 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-11T14:41:17.651 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-11T14:41:17.651 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_gc_processor_period": "3600", 2024-09-11T14:41:17.651 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-11T14:41:17.651 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-11T14:41:17.651 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-11T14:41:17.651 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-11T14:41:17.651 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_init_timeout": "300", 2024-09-11T14:41:17.651 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-11T14:41:17.651 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-11T14:41:17.652 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_kafka_connection_idle": "300", 2024-09-11T14:41:17.652 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-11T14:41:17.652 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-11T14:41:17.652 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-11T14:41:17.652 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-11T14:41:17.652 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-11T14:41:17.652 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_keystone_admin_domain": "", 2024-09-11T14:41:17.652 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_keystone_admin_password": "", 2024-09-11T14:41:17.652 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_keystone_admin_password_path": "", 2024-09-11T14:41:17.652 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_keystone_admin_project": "", 2024-09-11T14:41:17.652 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_keystone_admin_tenant": "", 2024-09-11T14:41:17.652 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_keystone_admin_token": "", 2024-09-11T14:41:17.652 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_keystone_admin_token_path": "", 2024-09-11T14:41:17.653 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_keystone_admin_user": "", 2024-09-11T14:41:17.653 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_keystone_barbican_domain": "", 2024-09-11T14:41:17.653 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_keystone_barbican_password": "", 2024-09-11T14:41:17.653 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_keystone_barbican_project": "", 2024-09-11T14:41:17.653 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-11T14:41:17.653 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_keystone_barbican_user": "", 2024-09-11T14:41:17.653 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-11T14:41:17.653 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-11T14:41:17.653 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-11T14:41:17.653 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-11T14:41:17.653 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-11T14:41:17.653 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_keystone_url": "", 2024-09-11T14:41:17.653 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-11T14:41:17.654 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_lc_debug_interval": "-1", 2024-09-11T14:41:17.654 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_lc_lock_max_time": "90", 2024-09-11T14:41:17.654 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_lc_max_objs": "32", 2024-09-11T14:41:17.654 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_lc_max_rules": "1000", 2024-09-11T14:41:17.654 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_lc_max_worker": "3", 2024-09-11T14:41:17.654 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-11T14:41:17.654 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_lc_thread_delay": "0", 2024-09-11T14:41:17.654 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-11T14:41:17.654 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_ldap_dnattr": "uid", 2024-09-11T14:41:17.654 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-11T14:41:17.654 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_ldap_searchfilter": "", 2024-09-11T14:41:17.654 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-11T14:41:17.654 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_ldap_uri": "", 2024-09-11T14:41:17.655 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_lfuda_sync_frequency": "60", 2024-09-11T14:41:17.655 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-11T14:41:17.655 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-11T14:41:17.655 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-11T14:41:17.655 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_log_http_headers": "", 2024-09-11T14:41:17.655 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-11T14:41:17.655 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-11T14:41:17.655 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_log_object_name_utc": "false", 2024-09-11T14:41:17.655 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-11T14:41:17.655 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-11T14:41:17.655 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_max_attr_name_len": "0", 2024-09-11T14:41:17.655 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_max_attr_size": "0", 2024-09-11T14:41:17.655 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-11T14:41:17.655 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_max_chunk_size": "4194304", 2024-09-11T14:41:17.656 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-11T14:41:17.656 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-11T14:41:17.656 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-11T14:41:17.656 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_max_listing_results": "5000", 2024-09-11T14:41:17.656 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_max_notify_retries": "10", 2024-09-11T14:41:17.656 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-11T14:41:17.656 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_max_put_param_size": "1048576", 2024-09-11T14:41:17.656 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_max_put_size": "5368709120", 2024-09-11T14:41:17.656 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_max_slo_entries": "1000", 2024-09-11T14:41:17.656 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_md_log_max_shards": "64", 2024-09-11T14:41:17.656 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-11T14:41:17.656 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-11T14:41:17.656 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-11T14:41:17.657 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-11T14:41:17.657 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_mp_lock_max_time": "600", 2024-09-11T14:41:17.657 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-11T14:41:17.657 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-11T14:41:17.657 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-11T14:41:17.657 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-11T14:41:17.657 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-11T14:41:17.657 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-11T14:41:17.657 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-11T14:41:17.657 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-11T14:41:17.657 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_nfs_max_gc": "300", 2024-09-11T14:41:17.657 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-11T14:41:17.657 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-11T14:41:17.657 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-11T14:41:17.658 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-11T14:41:17.658 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-11T14:41:17.658 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-11T14:41:17.658 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-11T14:41:17.658 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_num_async_rados_threads": "32", 2024-09-11T14:41:17.658 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_num_control_oids": "8", 2024-09-11T14:41:17.658 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_numa_node": "-1", 2024-09-11T14:41:17.658 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-11T14:41:17.658 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-11T14:41:17.658 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_objexp_chunk_size": "100", 2024-09-11T14:41:17.658 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_objexp_gc_interval": "600", 2024-09-11T14:41:17.658 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-11T14:41:17.658 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-11T14:41:17.658 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-11T14:41:17.659 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_op_thread_timeout": "600", 2024-09-11T14:41:17.659 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_op_tracing": "false", 2024-09-11T14:41:17.659 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_opa_token": "", 2024-09-11T14:41:17.659 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_opa_url": "", 2024-09-11T14:41:17.659 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_opa_verify_ssl": "true", 2024-09-11T14:41:17.659 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-11T14:41:17.659 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-11T14:41:17.659 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_ops_log_rados": "false", 2024-09-11T14:41:17.659 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_ops_log_socket_path": "", 2024-09-11T14:41:17.659 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-11T14:41:17.659 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-11T14:41:17.659 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-11T14:41:17.659 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_period_push_interval": "2.000000", 2024-09-11T14:41:17.660 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-11T14:41:17.660 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-11T14:41:17.660 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-11T14:41:17.660 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-11T14:41:17.660 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_posix_cache_lanes": "3", 2024-09-11T14:41:17.660 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-11T14:41:17.660 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-11T14:41:17.660 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_posix_cache_partitions": "3", 2024-09-11T14:41:17.660 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-11T14:41:17.660 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_print_continue": "true", 2024-09-11T14:41:17.660 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-11T14:41:17.660 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-11T14:41:17.660 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-11T14:41:17.661 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-11T14:41:17.661 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-11T14:41:17.661 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_rados_tracing": "false", 2024-09-11T14:41:17.661 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_realm": "", 2024-09-11T14:41:17.661 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_realm_id": "", 2024-09-11T14:41:17.661 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-11T14:41:17.661 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_region": "", 2024-09-11T14:41:17.661 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-11T14:41:17.661 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-11T14:41:17.661 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-11T14:41:17.661 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-11T14:41:17.661 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_request_uri": "", 2024-09-11T14:41:17.661 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_reshard_batch_size": "64", 2024-09-11T14:41:17.661 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-11T14:41:17.662 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_reshard_debug_interval": "-1", 2024-09-11T14:41:17.662 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_reshard_max_aio": "128", 2024-09-11T14:41:17.662 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_reshard_num_logs": "16", 2024-09-11T14:41:17.662 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_reshard_progress_judge_interval": "120", 2024-09-11T14:41:17.662 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_reshard_progress_judge_ratio": "0.500000", 2024-09-11T14:41:17.662 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_reshard_thread_interval": "600", 2024-09-11T14:41:17.662 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_reshardlog_threshold": "30000", 2024-09-11T14:41:17.662 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_resolve_cname": "false", 2024-09-11T14:41:17.662 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-11T14:41:17.662 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_run_sync_thread": "true", 2024-09-11T14:41:17.662 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-11T14:41:17.662 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-11T14:41:17.662 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-11T14:41:17.662 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-11T14:41:17.663 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-11T14:41:17.663 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-11T14:41:17.663 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-11T14:41:17.663 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-11T14:41:17.663 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-11T14:41:17.663 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_scheduler_type": "throttler", 2024-09-11T14:41:17.663 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_script_uri": "", 2024-09-11T14:41:17.663 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_service_provider_name": "", 2024-09-11T14:41:17.663 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-11T14:41:17.663 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_sts_client_id": "", 2024-09-11T14:41:17.663 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_sts_client_secret": "", 2024-09-11T14:41:17.663 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_sts_entry": "sts", 2024-09-11T14:41:17.663 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_sts_key": "sts", 2024-09-11T14:41:17.664 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-11T14:41:17.664 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_sts_min_session_duration": "900", 2024-09-11T14:41:17.664 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_sts_token_introspection_url": "", 2024-09-11T14:41:17.664 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_swift_account_in_url": "false", 2024-09-11T14:41:17.664 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_swift_auth_entry": "auth", 2024-09-11T14:41:17.664 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_swift_auth_url": "", 2024-09-11T14:41:17.664 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_swift_custom_header": "", 2024-09-11T14:41:17.664 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-11T14:41:17.664 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_swift_need_stats": "true", 2024-09-11T14:41:17.664 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_swift_tenant_name": "", 2024-09-11T14:41:17.664 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_swift_token_expiration": "86400", 2024-09-11T14:41:17.664 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_swift_url": "", 2024-09-11T14:41:17.664 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_swift_url_prefix": "swift", 2024-09-11T14:41:17.665 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-11T14:41:17.665 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-11T14:41:17.665 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-11T14:41:17.665 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_sync_lease_period": "120", 2024-09-11T14:41:17.665 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-11T14:41:17.665 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-11T14:41:17.665 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-11T14:41:17.665 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-11T14:41:17.665 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-11T14:41:17.665 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-11T14:41:17.665 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-11T14:41:17.665 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-11T14:41:17.665 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-11T14:41:17.666 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_thread_pool_size": "512", 2024-09-11T14:41:17.666 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-11T14:41:17.666 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-11T14:41:17.666 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-11T14:41:17.666 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-11T14:41:17.666 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_torrent_comment": "", 2024-09-11T14:41:17.666 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_torrent_createby": "", 2024-09-11T14:41:17.666 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_torrent_encoding": "", 2024-09-11T14:41:17.666 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_torrent_flag": "false", 2024-09-11T14:41:17.666 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-11T14:41:17.666 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_torrent_origin": "", 2024-09-11T14:41:17.666 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-11T14:41:17.666 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_torrent_tracker": "", 2024-09-11T14:41:17.666 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_trust_forwarded_https": "false", 2024-09-11T14:41:17.667 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-11T14:41:17.667 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-11T14:41:17.667 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_usage_max_shards": "32", 2024-09-11T14:41:17.667 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_usage_max_user_shards": "1", 2024-09-11T14:41:17.667 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_use_opa_authz": "false", 2024-09-11T14:41:17.667 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_user_counters_cache": "false", 2024-09-11T14:41:17.667 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-11T14:41:17.667 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-11T14:41:17.667 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-11T14:41:17.667 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_user_max_buckets": "1000", 2024-09-11T14:41:17.667 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_user_policies_max_num": "100", 2024-09-11T14:41:17.667 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-11T14:41:17.667 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-11T14:41:17.668 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-11T14:41:17.668 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-11T14:41:17.668 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_user_unique_email": "true", 2024-09-11T14:41:17.668 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_verify_ssl": "true", 2024-09-11T14:41:17.668 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-11T14:41:17.668 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_zone": "", 2024-09-11T14:41:17.668 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_zone_id": "", 2024-09-11T14:41:17.668 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-11T14:41:17.668 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_zonegroup": "", 2024-09-11T14:41:17.668 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_zonegroup_id": "", 2024-09-11T14:41:17.668 INFO:teuthology.orchestra.run.smithi181.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-11T14:41:17.668 INFO:teuthology.orchestra.run.smithi181.stdout: "rgwlc_auto_session_clear": "true", 2024-09-11T14:41:17.668 INFO:teuthology.orchestra.run.smithi181.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-11T14:41:17.668 INFO:teuthology.orchestra.run.smithi181.stdout: "rocksdb_block_size": "4096", 2024-09-11T14:41:17.669 INFO:teuthology.orchestra.run.smithi181.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-11T14:41:17.669 INFO:teuthology.orchestra.run.smithi181.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-11T14:41:17.669 INFO:teuthology.orchestra.run.smithi181.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-11T14:41:17.669 INFO:teuthology.orchestra.run.smithi181.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-11T14:41:17.669 INFO:teuthology.orchestra.run.smithi181.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-11T14:41:17.669 INFO:teuthology.orchestra.run.smithi181.stdout: "rocksdb_cache_size": "536870912", 2024-09-11T14:41:17.669 INFO:teuthology.orchestra.run.smithi181.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-11T14:41:17.669 INFO:teuthology.orchestra.run.smithi181.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-11T14:41:17.669 INFO:teuthology.orchestra.run.smithi181.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-11T14:41:17.669 INFO:teuthology.orchestra.run.smithi181.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-11T14:41:17.669 INFO:teuthology.orchestra.run.smithi181.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-11T14:41:17.669 INFO:teuthology.orchestra.run.smithi181.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-11T14:41:17.669 INFO:teuthology.orchestra.run.smithi181.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-11T14:41:17.669 INFO:teuthology.orchestra.run.smithi181.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-11T14:41:17.670 INFO:teuthology.orchestra.run.smithi181.stdout: "rocksdb_index_type": "binary_search", 2024-09-11T14:41:17.670 INFO:teuthology.orchestra.run.smithi181.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-11T14:41:17.670 INFO:teuthology.orchestra.run.smithi181.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-11T14:41:17.670 INFO:teuthology.orchestra.run.smithi181.stdout: "rocksdb_partition_filters": "false", 2024-09-11T14:41:17.670 INFO:teuthology.orchestra.run.smithi181.stdout: "rocksdb_perf": "false", 2024-09-11T14:41:17.670 INFO:teuthology.orchestra.run.smithi181.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-11T14:41:17.670 INFO:teuthology.orchestra.run.smithi181.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-11T14:41:17.670 INFO:teuthology.orchestra.run.smithi181.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-11T14:41:17.670 INFO:teuthology.orchestra.run.smithi181.stdout: "run_dir": "/var/run/ceph", 2024-09-11T14:41:17.670 INFO:teuthology.orchestra.run.smithi181.stdout: "seastore_block_create": "true", 2024-09-11T14:41:17.670 INFO:teuthology.orchestra.run.smithi181.stdout: "seastore_cache_lru_size": "67108864", 2024-09-11T14:41:17.670 INFO:teuthology.orchestra.run.smithi181.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-11T14:41:17.670 INFO:teuthology.orchestra.run.smithi181.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-11T14:41:17.671 INFO:teuthology.orchestra.run.smithi181.stdout: "seastore_default_max_object_size": "16777216", 2024-09-11T14:41:17.671 INFO:teuthology.orchestra.run.smithi181.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-11T14:41:17.671 INFO:teuthology.orchestra.run.smithi181.stdout: "seastore_device_size": "53687091200", 2024-09-11T14:41:17.671 INFO:teuthology.orchestra.run.smithi181.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-11T14:41:17.671 INFO:teuthology.orchestra.run.smithi181.stdout: "seastore_full_integrity_check": "false", 2024-09-11T14:41:17.671 INFO:teuthology.orchestra.run.smithi181.stdout: "seastore_journal_batch_capacity": "16", 2024-09-11T14:41:17.671 INFO:teuthology.orchestra.run.smithi181.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-11T14:41:17.671 INFO:teuthology.orchestra.run.smithi181.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-11T14:41:17.671 INFO:teuthology.orchestra.run.smithi181.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-11T14:41:17.671 INFO:teuthology.orchestra.run.smithi181.stdout: "seastore_main_device_type": "SSD", 2024-09-11T14:41:17.671 INFO:teuthology.orchestra.run.smithi181.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-11T14:41:17.671 INFO:teuthology.orchestra.run.smithi181.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-11T14:41:17.671 INFO:teuthology.orchestra.run.smithi181.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-11T14:41:17.671 INFO:teuthology.orchestra.run.smithi181.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-11T14:41:17.672 INFO:teuthology.orchestra.run.smithi181.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-11T14:41:17.672 INFO:teuthology.orchestra.run.smithi181.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-11T14:41:17.672 INFO:teuthology.orchestra.run.smithi181.stdout: "seastore_segment_size": "67108864", 2024-09-11T14:41:17.672 INFO:teuthology.orchestra.run.smithi181.stdout: "set_keepcaps": "false", 2024-09-11T14:41:17.672 INFO:teuthology.orchestra.run.smithi181.stdout: "setgroup": "", 2024-09-11T14:41:17.672 INFO:teuthology.orchestra.run.smithi181.stdout: "setuser": "", 2024-09-11T14:41:17.672 INFO:teuthology.orchestra.run.smithi181.stdout: "setuser_match_path": "", 2024-09-11T14:41:17.672 INFO:teuthology.orchestra.run.smithi181.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-11T14:41:17.672 INFO:teuthology.orchestra.run.smithi181.stdout: "thp": "false", 2024-09-11T14:41:17.672 INFO:teuthology.orchestra.run.smithi181.stdout: "threadpool_default_timeout": "60", 2024-09-11T14:41:17.672 INFO:teuthology.orchestra.run.smithi181.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-11T14:41:17.672 INFO:teuthology.orchestra.run.smithi181.stdout: "throttler_perf_counter": "true", 2024-09-11T14:41:17.672 INFO:teuthology.orchestra.run.smithi181.stdout: "tmp_dir": "/tmp", 2024-09-11T14:41:17.673 INFO:teuthology.orchestra.run.smithi181.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2024-09-11T14:41:17.673 INFO:teuthology.orchestra.run.smithi181.stdout: "uadk_compressor_enabled": "false", 2024-09-11T14:41:17.673 INFO:teuthology.orchestra.run.smithi181.stdout: "uadk_wd_sync_ctx_num": "2" 2024-09-11T14:41:17.673 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T14:41:17.675 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-11T14:41:17.780 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:17.777+0000 7fd496c74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:17.783 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:17.782+0000 7fd496c74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:17.797 INFO:teuthology.orchestra.run.smithi181.stdout:{} 2024-09-11T14:41:17.798 INFO:teuthology.orchestra.run.smithi181.stderr:mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-11T14:41:17.809 DEBUG:teuthology.orchestra.run.smithi203:> type ceph-objectstore-tool 2024-09-11T14:41:17.828 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-11T14:41:17.828 DEBUG:teuthology.orchestra.run.smithi181:> type ceph-objectstore-tool 2024-09-11T14:41:17.843 INFO:teuthology.orchestra.run.smithi181.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-11T14:41:17.843 INFO:teuthology.run_tasks:Running task radosbench... 2024-09-11T14:41:17.853 INFO:tasks.radosbench:Beginning radosbench... 2024-09-11T14:41:17.853 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-11T14:41:17.853 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2024-09-11T14:41:17.854 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-11T14:41:17.854 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:41:17.854 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:41:17.854 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-11T14:41:17.854 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-11T14:41:17.854 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-11T14:41:17.855 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:41:18.004 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:18.000+0000 7ff821991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:18.007 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:18.000+0000 7f17500a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:18.008 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:18.006+0000 7ff821991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:18.014 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:18.012+0000 7f17500a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:18.015 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:18.005+0000 7f2dc9991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:18.021 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:18.019+0000 7f2dc9991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:18.032 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:41:18.032 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:41:18.032 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:41:18.032 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:41:18.032 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:41:18.032 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:41:18.033 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:41:18.033 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:41:18.033 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:41:18.033 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:41:18.033 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:41:18.033 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:41:18.035 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:41:18.036 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:41:18.036 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-11T14:41:18.150 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:18.143+0000 7f6e56f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:18.153 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:18.152+0000 7f6e56f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:18.171 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:41:18.171 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:41:18.171 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:41:18.171 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:41:18.171 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:41:18.171 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:41:18.172 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:41:18.172 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:41:18.172 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:41:18.172 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:41:18.172 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:41:18.172 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:41:18.174 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:41:18.175 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:41:18.175 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-11T14:41:18.323 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:18.317+0000 7f03bfe57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:18.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:18.326+0000 7f03bfe57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:18.345 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:41:18.346 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:41:18.346 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:41:18.346 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:41:18.346 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:41:18.346 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:41:18.346 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:41:18.346 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:41:18.346 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:41:18.346 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:41:18.346 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:41:18.346 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:41:18.349 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:41:18.349 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:41:18.349 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-11T14:41:18.458 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:18.452+0000 7f1d5bf91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:18.462 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:18.461+0000 7f1d5bf91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:18.477 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:41:18.478 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:41:18.478 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:41:18.478 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:41:18.478 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:41:18.478 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:41:18.478 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:41:18.478 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:41:18.478 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:41:18.478 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:41:18.478 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:41:18.478 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:41:18.480 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:41:18.481 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:41:18.811 INFO:teuthology.orchestra.run.smithi181.stderr:pool 'unique_pool_0' created 2024-09-11T14:41:18.829 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-11T14:41:18.985 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:18.974+0000 7fe1ba951640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:18.989 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:18.988+0000 7fe1ba951640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:19.812 INFO:teuthology.orchestra.run.smithi181.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-11T14:41:20.125 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:41:20.232 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:20.229+0000 7f6b6f711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:20.236 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:20.234+0000 7f6b6f711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:20.832 DEBUG:teuthology.orchestra.run.smithi181:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 150 write' 2024-09-11T14:41:20.834 INFO:teuthology.run_tasks:Running task admin_socket... 2024-09-11T14:41:20.846 DEBUG:tasks.admin_socket:Running admin socket tests on client.0 2024-09-11T14:41:20.846 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -- /home/ubuntu/cephtest/admin_socket_client.0 && while test '!' -e /var/run/ceph/ceph-client.0.asok ; do sleep 1 ; done 2024-09-11T14:41:20.903 INFO:tasks.radosbench.radosbench.0.smithi181.stdout:hints = 1 2024-09-11T14:41:20.903 INFO:tasks.radosbench.radosbench.0.smithi181.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 150 seconds or 0 objects 2024-09-11T14:41:20.903 INFO:tasks.radosbench.radosbench.0.smithi181.stdout:Object prefix: benchmark_data_smithi181_30163 2024-09-11T14:41:20.903 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T14:41:20.903 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 0 0 0 0 0 0 - 0 2024-09-11T14:41:21.866 DEBUG:tasks.admin_socket:Testing objecter_requests with config {'test': 'http://git.ceph.com/?p={repo};a=blob_plain;f=src/test/admin_socket/objecter_requests;hb={branch}', 'branch': 'main'} 2024-09-11T14:41:21.866 DEBUG:teuthology.orchestra.run.smithi181:> wget -q -O /home/ubuntu/cephtest/admin_socket_client.0/objecter_requests -- 'http://git.ceph.com/?p=ceph-ci.git;a=blob_plain;f=src/test/admin_socket/objecter_requests;hb=main' && chmod u=rx -- /home/ubuntu/cephtest/admin_socket_client.0/objecter_requests 2024-09-11T14:41:21.903 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 1 16 82 66 4.12476 4.125 0.418808 0.178521 2024-09-11T14:41:22.283 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-client.0.asok objecter_requests 2024-09-11T14:41:22.334 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:41:22.442 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T14:41:22.442 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-11T14:41:22.443 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T14:41:22.443 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 80, 2024-09-11T14:41:22.443 INFO:teuthology.orchestra.run.smithi181.stdout: "pg": "3.3adab0dc", 2024-09-11T14:41:22.443 INFO:teuthology.orchestra.run.smithi181.stdout: "osd": 0, 2024-09-11T14:41:22.443 INFO:teuthology.orchestra.run.smithi181.stdout: "object_id": "benchmark_data_smithi181_30163_object79", 2024-09-11T14:41:22.443 INFO:teuthology.orchestra.run.smithi181.stdout: "object_locator": "@3", 2024-09-11T14:41:22.443 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_id": "benchmark_data_smithi181_30163_object79", 2024-09-11T14:41:22.443 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_locator": "@3", 2024-09-11T14:41:22.443 INFO:teuthology.orchestra.run.smithi181.stdout: "paused": 0, 2024-09-11T14:41:22.443 INFO:teuthology.orchestra.run.smithi181.stdout: "used_replica": 0, 2024-09-11T14:41:22.443 INFO:teuthology.orchestra.run.smithi181.stdout: "precalc_pgid": 0, 2024-09-11T14:41:22.443 INFO:teuthology.orchestra.run.smithi181.stdout: "last_sent": "484.666112s", 2024-09-11T14:41:22.443 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.552989179, 2024-09-11T14:41:22.444 INFO:teuthology.orchestra.run.smithi181.stdout: "attempts": 1, 2024-09-11T14:41:22.444 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": "head", 2024-09-11T14:41:22.444 INFO:teuthology.orchestra.run.smithi181.stdout: "snap_context": "0=[]", 2024-09-11T14:41:22.444 INFO:teuthology.orchestra.run.smithi181.stdout: "mtime": "2024-09-11T14:41:21.888650+0000", 2024-09-11T14:41:22.444 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_ops": [ 2024-09-11T14:41:22.444 INFO:teuthology.orchestra.run.smithi181.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-11T14:41:22.444 INFO:teuthology.orchestra.run.smithi181.stdout: "write 0~65536 in=65536b" 2024-09-11T14:41:22.444 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T14:41:22.444 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T14:41:22.444 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T14:41:22.444 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 83, 2024-09-11T14:41:22.444 INFO:teuthology.orchestra.run.smithi181.stdout: "pg": "3.a9f591bc", 2024-09-11T14:41:22.444 INFO:teuthology.orchestra.run.smithi181.stdout: "osd": 0, 2024-09-11T14:41:22.445 INFO:teuthology.orchestra.run.smithi181.stdout: "object_id": "benchmark_data_smithi181_30163_object82", 2024-09-11T14:41:22.445 INFO:teuthology.orchestra.run.smithi181.stdout: "object_locator": "@3", 2024-09-11T14:41:22.445 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_id": "benchmark_data_smithi181_30163_object82", 2024-09-11T14:41:22.445 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_locator": "@3", 2024-09-11T14:41:22.445 INFO:teuthology.orchestra.run.smithi181.stdout: "paused": 0, 2024-09-11T14:41:22.445 INFO:teuthology.orchestra.run.smithi181.stdout: "used_replica": 0, 2024-09-11T14:41:22.445 INFO:teuthology.orchestra.run.smithi181.stdout: "precalc_pgid": 0, 2024-09-11T14:41:22.445 INFO:teuthology.orchestra.run.smithi181.stdout: "last_sent": "484.698112s", 2024-09-11T14:41:22.445 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.52098980500000003, 2024-09-11T14:41:22.445 INFO:teuthology.orchestra.run.smithi181.stdout: "attempts": 1, 2024-09-11T14:41:22.445 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": "head", 2024-09-11T14:41:22.445 INFO:teuthology.orchestra.run.smithi181.stdout: "snap_context": "0=[]", 2024-09-11T14:41:22.445 INFO:teuthology.orchestra.run.smithi181.stdout: "mtime": "2024-09-11T14:41:21.920250+0000", 2024-09-11T14:41:22.446 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_ops": [ 2024-09-11T14:41:22.446 INFO:teuthology.orchestra.run.smithi181.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-11T14:41:22.446 INFO:teuthology.orchestra.run.smithi181.stdout: "write 0~65536 in=65536b" 2024-09-11T14:41:22.446 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T14:41:22.446 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T14:41:22.446 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T14:41:22.446 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 91, 2024-09-11T14:41:22.446 INFO:teuthology.orchestra.run.smithi181.stdout: "pg": "3.b53b711f", 2024-09-11T14:41:22.446 INFO:teuthology.orchestra.run.smithi181.stdout: "osd": 0, 2024-09-11T14:41:22.446 INFO:teuthology.orchestra.run.smithi181.stdout: "object_id": "benchmark_data_smithi181_30163_object90", 2024-09-11T14:41:22.446 INFO:teuthology.orchestra.run.smithi181.stdout: "object_locator": "@3", 2024-09-11T14:41:22.446 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_id": "benchmark_data_smithi181_30163_object90", 2024-09-11T14:41:22.446 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_locator": "@3", 2024-09-11T14:41:22.446 INFO:teuthology.orchestra.run.smithi181.stdout: "paused": 0, 2024-09-11T14:41:22.447 INFO:teuthology.orchestra.run.smithi181.stdout: "used_replica": 0, 2024-09-11T14:41:22.447 INFO:teuthology.orchestra.run.smithi181.stdout: "precalc_pgid": 0, 2024-09-11T14:41:22.447 INFO:teuthology.orchestra.run.smithi181.stdout: "last_sent": "484.901108s", 2024-09-11T14:41:22.447 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.317993778, 2024-09-11T14:41:22.447 INFO:teuthology.orchestra.run.smithi181.stdout: "attempts": 1, 2024-09-11T14:41:22.447 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": "head", 2024-09-11T14:41:22.447 INFO:teuthology.orchestra.run.smithi181.stdout: "snap_context": "0=[]", 2024-09-11T14:41:22.447 INFO:teuthology.orchestra.run.smithi181.stdout: "mtime": "2024-09-11T14:41:22.123362+0000", 2024-09-11T14:41:22.447 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_ops": [ 2024-09-11T14:41:22.447 INFO:teuthology.orchestra.run.smithi181.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-11T14:41:22.447 INFO:teuthology.orchestra.run.smithi181.stdout: "write 0~65536 in=65536b" 2024-09-11T14:41:22.447 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T14:41:22.447 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T14:41:22.448 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T14:41:22.448 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 107, 2024-09-11T14:41:22.448 INFO:teuthology.orchestra.run.smithi181.stdout: "pg": "3.5b0c460c", 2024-09-11T14:41:22.448 INFO:teuthology.orchestra.run.smithi181.stdout: "osd": 0, 2024-09-11T14:41:22.448 INFO:teuthology.orchestra.run.smithi181.stdout: "object_id": "benchmark_data_smithi181_30163_object106", 2024-09-11T14:41:22.448 INFO:teuthology.orchestra.run.smithi181.stdout: "object_locator": "@3", 2024-09-11T14:41:22.448 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_id": "benchmark_data_smithi181_30163_object106", 2024-09-11T14:41:22.448 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_locator": "@3", 2024-09-11T14:41:22.448 INFO:teuthology.orchestra.run.smithi181.stdout: "paused": 0, 2024-09-11T14:41:22.448 INFO:teuthology.orchestra.run.smithi181.stdout: "used_replica": 0, 2024-09-11T14:41:22.448 INFO:teuthology.orchestra.run.smithi181.stdout: "precalc_pgid": 0, 2024-09-11T14:41:22.448 INFO:teuthology.orchestra.run.smithi181.stdout: "last_sent": "485.126103s", 2024-09-11T14:41:22.448 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.09299818, 2024-09-11T14:41:22.449 INFO:teuthology.orchestra.run.smithi181.stdout: "attempts": 1, 2024-09-11T14:41:22.449 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": "head", 2024-09-11T14:41:22.449 INFO:teuthology.orchestra.run.smithi181.stdout: "snap_context": "0=[]", 2024-09-11T14:41:22.449 INFO:teuthology.orchestra.run.smithi181.stdout: "mtime": "2024-09-11T14:41:22.348070+0000", 2024-09-11T14:41:22.449 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_ops": [ 2024-09-11T14:41:22.449 INFO:teuthology.orchestra.run.smithi181.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-11T14:41:22.449 INFO:teuthology.orchestra.run.smithi181.stdout: "write 0~65536 in=65536b" 2024-09-11T14:41:22.449 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T14:41:22.449 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T14:41:22.449 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T14:41:22.449 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 110, 2024-09-11T14:41:22.449 INFO:teuthology.orchestra.run.smithi181.stdout: "pg": "3.1f868bbc", 2024-09-11T14:41:22.449 INFO:teuthology.orchestra.run.smithi181.stdout: "osd": 0, 2024-09-11T14:41:22.449 INFO:teuthology.orchestra.run.smithi181.stdout: "object_id": "benchmark_data_smithi181_30163_object109", 2024-09-11T14:41:22.450 INFO:teuthology.orchestra.run.smithi181.stdout: "object_locator": "@3", 2024-09-11T14:41:22.450 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_id": "benchmark_data_smithi181_30163_object109", 2024-09-11T14:41:22.450 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_locator": "@3", 2024-09-11T14:41:22.450 INFO:teuthology.orchestra.run.smithi181.stdout: "paused": 0, 2024-09-11T14:41:22.450 INFO:teuthology.orchestra.run.smithi181.stdout: "used_replica": 0, 2024-09-11T14:41:22.450 INFO:teuthology.orchestra.run.smithi181.stdout: "precalc_pgid": 0, 2024-09-11T14:41:22.450 INFO:teuthology.orchestra.run.smithi181.stdout: "last_sent": "485.191102s", 2024-09-11T14:41:22.450 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.027999452000000001, 2024-09-11T14:41:22.450 INFO:teuthology.orchestra.run.smithi181.stdout: "attempts": 1, 2024-09-11T14:41:22.450 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": "head", 2024-09-11T14:41:22.450 INFO:teuthology.orchestra.run.smithi181.stdout: "snap_context": "0=[]", 2024-09-11T14:41:22.450 INFO:teuthology.orchestra.run.smithi181.stdout: "mtime": "2024-09-11T14:41:22.413583+0000", 2024-09-11T14:41:22.450 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_ops": [ 2024-09-11T14:41:22.450 INFO:teuthology.orchestra.run.smithi181.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-11T14:41:22.451 INFO:teuthology.orchestra.run.smithi181.stdout: "write 0~65536 in=65536b" 2024-09-11T14:41:22.451 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T14:41:22.451 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T14:41:22.451 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T14:41:22.451 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 76, 2024-09-11T14:41:22.451 INFO:teuthology.orchestra.run.smithi181.stdout: "pg": "3.89fcc6f2", 2024-09-11T14:41:22.451 INFO:teuthology.orchestra.run.smithi181.stdout: "osd": 1, 2024-09-11T14:41:22.451 INFO:teuthology.orchestra.run.smithi181.stdout: "object_id": "benchmark_data_smithi181_30163_object75", 2024-09-11T14:41:22.451 INFO:teuthology.orchestra.run.smithi181.stdout: "object_locator": "@3", 2024-09-11T14:41:22.451 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_id": "benchmark_data_smithi181_30163_object75", 2024-09-11T14:41:22.451 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_locator": "@3", 2024-09-11T14:41:22.451 INFO:teuthology.orchestra.run.smithi181.stdout: "paused": 0, 2024-09-11T14:41:22.451 INFO:teuthology.orchestra.run.smithi181.stdout: "used_replica": 0, 2024-09-11T14:41:22.452 INFO:teuthology.orchestra.run.smithi181.stdout: "precalc_pgid": 0, 2024-09-11T14:41:22.452 INFO:teuthology.orchestra.run.smithi181.stdout: "last_sent": "484.575114s", 2024-09-11T14:41:22.452 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.64398739900000002, 2024-09-11T14:41:22.452 INFO:teuthology.orchestra.run.smithi181.stdout: "attempts": 1, 2024-09-11T14:41:22.452 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": "head", 2024-09-11T14:41:22.452 INFO:teuthology.orchestra.run.smithi181.stdout: "snap_context": "0=[]", 2024-09-11T14:41:22.452 INFO:teuthology.orchestra.run.smithi181.stdout: "mtime": "2024-09-11T14:41:21.797504+0000", 2024-09-11T14:41:22.452 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_ops": [ 2024-09-11T14:41:22.452 INFO:teuthology.orchestra.run.smithi181.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-11T14:41:22.452 INFO:teuthology.orchestra.run.smithi181.stdout: "write 0~65536 in=65536b" 2024-09-11T14:41:22.452 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T14:41:22.452 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T14:41:22.452 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T14:41:22.453 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 93, 2024-09-11T14:41:22.453 INFO:teuthology.orchestra.run.smithi181.stdout: "pg": "3.8dd1fb5d", 2024-09-11T14:41:22.453 INFO:teuthology.orchestra.run.smithi181.stdout: "osd": 1, 2024-09-11T14:41:22.453 INFO:teuthology.orchestra.run.smithi181.stdout: "object_id": "benchmark_data_smithi181_30163_object92", 2024-09-11T14:41:22.453 INFO:teuthology.orchestra.run.smithi181.stdout: "object_locator": "@3", 2024-09-11T14:41:22.453 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_id": "benchmark_data_smithi181_30163_object92", 2024-09-11T14:41:22.453 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_locator": "@3", 2024-09-11T14:41:22.453 INFO:teuthology.orchestra.run.smithi181.stdout: "paused": 0, 2024-09-11T14:41:22.453 INFO:teuthology.orchestra.run.smithi181.stdout: "used_replica": 0, 2024-09-11T14:41:22.453 INFO:teuthology.orchestra.run.smithi181.stdout: "precalc_pgid": 0, 2024-09-11T14:41:22.453 INFO:teuthology.orchestra.run.smithi181.stdout: "last_sent": "484.938107s", 2024-09-11T14:41:22.453 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.28099450199999998, 2024-09-11T14:41:22.453 INFO:teuthology.orchestra.run.smithi181.stdout: "attempts": 1, 2024-09-11T14:41:22.454 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": "head", 2024-09-11T14:41:22.454 INFO:teuthology.orchestra.run.smithi181.stdout: "snap_context": "0=[]", 2024-09-11T14:41:22.454 INFO:teuthology.orchestra.run.smithi181.stdout: "mtime": "2024-09-11T14:41:22.160732+0000", 2024-09-11T14:41:22.454 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_ops": [ 2024-09-11T14:41:22.454 INFO:teuthology.orchestra.run.smithi181.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-11T14:41:22.454 INFO:teuthology.orchestra.run.smithi181.stdout: "write 0~65536 in=65536b" 2024-09-11T14:41:22.454 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T14:41:22.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T14:41:22.454 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T14:41:22.454 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 94, 2024-09-11T14:41:22.454 INFO:teuthology.orchestra.run.smithi181.stdout: "pg": "3.6da523d5", 2024-09-11T14:41:22.454 INFO:teuthology.orchestra.run.smithi181.stdout: "osd": 1, 2024-09-11T14:41:22.454 INFO:teuthology.orchestra.run.smithi181.stdout: "object_id": "benchmark_data_smithi181_30163_object93", 2024-09-11T14:41:22.455 INFO:teuthology.orchestra.run.smithi181.stdout: "object_locator": "@3", 2024-09-11T14:41:22.455 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_id": "benchmark_data_smithi181_30163_object93", 2024-09-11T14:41:22.455 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_locator": "@3", 2024-09-11T14:41:22.455 INFO:teuthology.orchestra.run.smithi181.stdout: "paused": 0, 2024-09-11T14:41:22.455 INFO:teuthology.orchestra.run.smithi181.stdout: "used_replica": 0, 2024-09-11T14:41:22.455 INFO:teuthology.orchestra.run.smithi181.stdout: "precalc_pgid": 0, 2024-09-11T14:41:22.455 INFO:teuthology.orchestra.run.smithi181.stdout: "last_sent": "484.944107s", 2024-09-11T14:41:22.455 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.274994619, 2024-09-11T14:41:22.455 INFO:teuthology.orchestra.run.smithi181.stdout: "attempts": 1, 2024-09-11T14:41:22.455 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": "head", 2024-09-11T14:41:22.455 INFO:teuthology.orchestra.run.smithi181.stdout: "snap_context": "0=[]", 2024-09-11T14:41:22.455 INFO:teuthology.orchestra.run.smithi181.stdout: "mtime": "2024-09-11T14:41:22.166456+0000", 2024-09-11T14:41:22.455 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_ops": [ 2024-09-11T14:41:22.456 INFO:teuthology.orchestra.run.smithi181.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-11T14:41:22.456 INFO:teuthology.orchestra.run.smithi181.stdout: "write 0~65536 in=65536b" 2024-09-11T14:41:22.456 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T14:41:22.456 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T14:41:22.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T14:41:22.456 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 96, 2024-09-11T14:41:22.456 INFO:teuthology.orchestra.run.smithi181.stdout: "pg": "3.5062bc82", 2024-09-11T14:41:22.456 INFO:teuthology.orchestra.run.smithi181.stdout: "osd": 1, 2024-09-11T14:41:22.456 INFO:teuthology.orchestra.run.smithi181.stdout: "object_id": "benchmark_data_smithi181_30163_object95", 2024-09-11T14:41:22.456 INFO:teuthology.orchestra.run.smithi181.stdout: "object_locator": "@3", 2024-09-11T14:41:22.456 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_id": "benchmark_data_smithi181_30163_object95", 2024-09-11T14:41:22.456 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_locator": "@3", 2024-09-11T14:41:22.456 INFO:teuthology.orchestra.run.smithi181.stdout: "paused": 0, 2024-09-11T14:41:22.457 INFO:teuthology.orchestra.run.smithi181.stdout: "used_replica": 0, 2024-09-11T14:41:22.457 INFO:teuthology.orchestra.run.smithi181.stdout: "precalc_pgid": 0, 2024-09-11T14:41:22.457 INFO:teuthology.orchestra.run.smithi181.stdout: "last_sent": "484.987106s", 2024-09-11T14:41:22.457 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.23199545999999999, 2024-09-11T14:41:22.457 INFO:teuthology.orchestra.run.smithi181.stdout: "attempts": 1, 2024-09-11T14:41:22.457 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": "head", 2024-09-11T14:41:22.457 INFO:teuthology.orchestra.run.smithi181.stdout: "snap_context": "0=[]", 2024-09-11T14:41:22.457 INFO:teuthology.orchestra.run.smithi181.stdout: "mtime": "2024-09-11T14:41:22.209644+0000", 2024-09-11T14:41:22.457 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_ops": [ 2024-09-11T14:41:22.457 INFO:teuthology.orchestra.run.smithi181.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-11T14:41:22.457 INFO:teuthology.orchestra.run.smithi181.stdout: "write 0~65536 in=65536b" 2024-09-11T14:41:22.457 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T14:41:22.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T14:41:22.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T14:41:22.458 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 101, 2024-09-11T14:41:22.458 INFO:teuthology.orchestra.run.smithi181.stdout: "pg": "3.ce43f5c5", 2024-09-11T14:41:22.458 INFO:teuthology.orchestra.run.smithi181.stdout: "osd": 1, 2024-09-11T14:41:22.458 INFO:teuthology.orchestra.run.smithi181.stdout: "object_id": "benchmark_data_smithi181_30163_object100", 2024-09-11T14:41:22.458 INFO:teuthology.orchestra.run.smithi181.stdout: "object_locator": "@3", 2024-09-11T14:41:22.458 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_id": "benchmark_data_smithi181_30163_object100", 2024-09-11T14:41:22.458 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_locator": "@3", 2024-09-11T14:41:22.458 INFO:teuthology.orchestra.run.smithi181.stdout: "paused": 0, 2024-09-11T14:41:22.458 INFO:teuthology.orchestra.run.smithi181.stdout: "used_replica": 0, 2024-09-11T14:41:22.458 INFO:teuthology.orchestra.run.smithi181.stdout: "precalc_pgid": 0, 2024-09-11T14:41:22.458 INFO:teuthology.orchestra.run.smithi181.stdout: "last_sent": "485.074104s", 2024-09-11T14:41:22.458 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.14499716300000001, 2024-09-11T14:41:22.459 INFO:teuthology.orchestra.run.smithi181.stdout: "attempts": 1, 2024-09-11T14:41:22.459 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": "head", 2024-09-11T14:41:22.459 INFO:teuthology.orchestra.run.smithi181.stdout: "snap_context": "0=[]", 2024-09-11T14:41:22.459 INFO:teuthology.orchestra.run.smithi181.stdout: "mtime": "2024-09-11T14:41:22.296638+0000", 2024-09-11T14:41:22.459 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_ops": [ 2024-09-11T14:41:22.459 INFO:teuthology.orchestra.run.smithi181.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-11T14:41:22.459 INFO:teuthology.orchestra.run.smithi181.stdout: "write 0~65536 in=65536b" 2024-09-11T14:41:22.459 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T14:41:22.459 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T14:41:22.459 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T14:41:22.459 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 103, 2024-09-11T14:41:22.459 INFO:teuthology.orchestra.run.smithi181.stdout: "pg": "3.1120432d", 2024-09-11T14:41:22.459 INFO:teuthology.orchestra.run.smithi181.stdout: "osd": 1, 2024-09-11T14:41:22.459 INFO:teuthology.orchestra.run.smithi181.stdout: "object_id": "benchmark_data_smithi181_30163_object102", 2024-09-11T14:41:22.460 INFO:teuthology.orchestra.run.smithi181.stdout: "object_locator": "@3", 2024-09-11T14:41:22.460 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_id": "benchmark_data_smithi181_30163_object102", 2024-09-11T14:41:22.460 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_locator": "@3", 2024-09-11T14:41:22.460 INFO:teuthology.orchestra.run.smithi181.stdout: "paused": 0, 2024-09-11T14:41:22.460 INFO:teuthology.orchestra.run.smithi181.stdout: "used_replica": 0, 2024-09-11T14:41:22.460 INFO:teuthology.orchestra.run.smithi181.stdout: "precalc_pgid": 0, 2024-09-11T14:41:22.460 INFO:teuthology.orchestra.run.smithi181.stdout: "last_sent": "485.104104s", 2024-09-11T14:41:22.460 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.11499775, 2024-09-11T14:41:22.460 INFO:teuthology.orchestra.run.smithi181.stdout: "attempts": 1, 2024-09-11T14:41:22.460 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": "head", 2024-09-11T14:41:22.460 INFO:teuthology.orchestra.run.smithi181.stdout: "snap_context": "0=[]", 2024-09-11T14:41:22.460 INFO:teuthology.orchestra.run.smithi181.stdout: "mtime": "2024-09-11T14:41:22.325985+0000", 2024-09-11T14:41:22.460 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_ops": [ 2024-09-11T14:41:22.461 INFO:teuthology.orchestra.run.smithi181.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-11T14:41:22.461 INFO:teuthology.orchestra.run.smithi181.stdout: "write 0~65536 in=65536b" 2024-09-11T14:41:22.461 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T14:41:22.461 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T14:41:22.461 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T14:41:22.461 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 108, 2024-09-11T14:41:22.461 INFO:teuthology.orchestra.run.smithi181.stdout: "pg": "3.baab5292", 2024-09-11T14:41:22.461 INFO:teuthology.orchestra.run.smithi181.stdout: "osd": 1, 2024-09-11T14:41:22.461 INFO:teuthology.orchestra.run.smithi181.stdout: "object_id": "benchmark_data_smithi181_30163_object107", 2024-09-11T14:41:22.461 INFO:teuthology.orchestra.run.smithi181.stdout: "object_locator": "@3", 2024-09-11T14:41:22.461 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_id": "benchmark_data_smithi181_30163_object107", 2024-09-11T14:41:22.461 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_locator": "@3", 2024-09-11T14:41:22.461 INFO:teuthology.orchestra.run.smithi181.stdout: "paused": 0, 2024-09-11T14:41:22.462 INFO:teuthology.orchestra.run.smithi181.stdout: "used_replica": 0, 2024-09-11T14:41:22.462 INFO:teuthology.orchestra.run.smithi181.stdout: "precalc_pgid": 0, 2024-09-11T14:41:22.462 INFO:teuthology.orchestra.run.smithi181.stdout: "last_sent": "485.162103s", 2024-09-11T14:41:22.462 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.056998884999999999, 2024-09-11T14:41:22.462 INFO:teuthology.orchestra.run.smithi181.stdout: "attempts": 1, 2024-09-11T14:41:22.462 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": "head", 2024-09-11T14:41:22.462 INFO:teuthology.orchestra.run.smithi181.stdout: "snap_context": "0=[]", 2024-09-11T14:41:22.462 INFO:teuthology.orchestra.run.smithi181.stdout: "mtime": "2024-09-11T14:41:22.384818+0000", 2024-09-11T14:41:22.462 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_ops": [ 2024-09-11T14:41:22.462 INFO:teuthology.orchestra.run.smithi181.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-11T14:41:22.462 INFO:teuthology.orchestra.run.smithi181.stdout: "write 0~65536 in=65536b" 2024-09-11T14:41:22.462 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T14:41:22.462 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T14:41:22.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T14:41:22.463 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 105, 2024-09-11T14:41:22.463 INFO:teuthology.orchestra.run.smithi181.stdout: "pg": "3.a6e91216", 2024-09-11T14:41:22.463 INFO:teuthology.orchestra.run.smithi181.stdout: "osd": 2, 2024-09-11T14:41:22.463 INFO:teuthology.orchestra.run.smithi181.stdout: "object_id": "benchmark_data_smithi181_30163_object104", 2024-09-11T14:41:22.463 INFO:teuthology.orchestra.run.smithi181.stdout: "object_locator": "@3", 2024-09-11T14:41:22.463 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_id": "benchmark_data_smithi181_30163_object104", 2024-09-11T14:41:22.463 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_locator": "@3", 2024-09-11T14:41:22.463 INFO:teuthology.orchestra.run.smithi181.stdout: "paused": 0, 2024-09-11T14:41:22.463 INFO:teuthology.orchestra.run.smithi181.stdout: "used_replica": 0, 2024-09-11T14:41:22.463 INFO:teuthology.orchestra.run.smithi181.stdout: "precalc_pgid": 0, 2024-09-11T14:41:22.463 INFO:teuthology.orchestra.run.smithi181.stdout: "last_sent": "485.114104s", 2024-09-11T14:41:22.463 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.104997945, 2024-09-11T14:41:22.463 INFO:teuthology.orchestra.run.smithi181.stdout: "attempts": 1, 2024-09-11T14:41:22.464 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": "head", 2024-09-11T14:41:22.464 INFO:teuthology.orchestra.run.smithi181.stdout: "snap_context": "0=[]", 2024-09-11T14:41:22.464 INFO:teuthology.orchestra.run.smithi181.stdout: "mtime": "2024-09-11T14:41:22.336182+0000", 2024-09-11T14:41:22.464 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_ops": [ 2024-09-11T14:41:22.464 INFO:teuthology.orchestra.run.smithi181.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-11T14:41:22.464 INFO:teuthology.orchestra.run.smithi181.stdout: "write 0~65536 in=65536b" 2024-09-11T14:41:22.464 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T14:41:22.464 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T14:41:22.464 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T14:41:22.464 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 109, 2024-09-11T14:41:22.464 INFO:teuthology.orchestra.run.smithi181.stdout: "pg": "3.ab15e626", 2024-09-11T14:41:22.464 INFO:teuthology.orchestra.run.smithi181.stdout: "osd": 2, 2024-09-11T14:41:22.464 INFO:teuthology.orchestra.run.smithi181.stdout: "object_id": "benchmark_data_smithi181_30163_object108", 2024-09-11T14:41:22.464 INFO:teuthology.orchestra.run.smithi181.stdout: "object_locator": "@3", 2024-09-11T14:41:22.465 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_id": "benchmark_data_smithi181_30163_object108", 2024-09-11T14:41:22.465 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_locator": "@3", 2024-09-11T14:41:22.465 INFO:teuthology.orchestra.run.smithi181.stdout: "paused": 0, 2024-09-11T14:41:22.465 INFO:teuthology.orchestra.run.smithi181.stdout: "used_replica": 0, 2024-09-11T14:41:22.465 INFO:teuthology.orchestra.run.smithi181.stdout: "precalc_pgid": 0, 2024-09-11T14:41:22.465 INFO:teuthology.orchestra.run.smithi181.stdout: "last_sent": "485.178102s", 2024-09-11T14:41:22.465 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.040999198000000001, 2024-09-11T14:41:22.465 INFO:teuthology.orchestra.run.smithi181.stdout: "attempts": 1, 2024-09-11T14:41:22.465 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": "head", 2024-09-11T14:41:22.465 INFO:teuthology.orchestra.run.smithi181.stdout: "snap_context": "0=[]", 2024-09-11T14:41:22.465 INFO:teuthology.orchestra.run.smithi181.stdout: "mtime": "2024-09-11T14:41:22.400323+0000", 2024-09-11T14:41:22.465 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_ops": [ 2024-09-11T14:41:22.466 INFO:teuthology.orchestra.run.smithi181.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-11T14:41:22.466 INFO:teuthology.orchestra.run.smithi181.stdout: "write 0~65536 in=65536b" 2024-09-11T14:41:22.466 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T14:41:22.466 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T14:41:22.466 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T14:41:22.466 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 106, 2024-09-11T14:41:22.466 INFO:teuthology.orchestra.run.smithi181.stdout: "pg": "3.d1b3a230", 2024-09-11T14:41:22.466 INFO:teuthology.orchestra.run.smithi181.stdout: "osd": 3, 2024-09-11T14:41:22.466 INFO:teuthology.orchestra.run.smithi181.stdout: "object_id": "benchmark_data_smithi181_30163_object105", 2024-09-11T14:41:22.466 INFO:teuthology.orchestra.run.smithi181.stdout: "object_locator": "@3", 2024-09-11T14:41:22.466 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_id": "benchmark_data_smithi181_30163_object105", 2024-09-11T14:41:22.466 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_locator": "@3", 2024-09-11T14:41:22.466 INFO:teuthology.orchestra.run.smithi181.stdout: "paused": 0, 2024-09-11T14:41:22.466 INFO:teuthology.orchestra.run.smithi181.stdout: "used_replica": 0, 2024-09-11T14:41:22.466 INFO:teuthology.orchestra.run.smithi181.stdout: "precalc_pgid": 0, 2024-09-11T14:41:22.467 INFO:teuthology.orchestra.run.smithi181.stdout: "last_sent": "485.114104s", 2024-09-11T14:41:22.467 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.104997945, 2024-09-11T14:41:22.467 INFO:teuthology.orchestra.run.smithi181.stdout: "attempts": 1, 2024-09-11T14:41:22.467 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": "head", 2024-09-11T14:41:22.467 INFO:teuthology.orchestra.run.smithi181.stdout: "snap_context": "0=[]", 2024-09-11T14:41:22.467 INFO:teuthology.orchestra.run.smithi181.stdout: "mtime": "2024-09-11T14:41:22.336226+0000", 2024-09-11T14:41:22.467 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_ops": [ 2024-09-11T14:41:22.467 INFO:teuthology.orchestra.run.smithi181.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-11T14:41:22.467 INFO:teuthology.orchestra.run.smithi181.stdout: "write 0~65536 in=65536b" 2024-09-11T14:41:22.467 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T14:41:22.467 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-11T14:41:22.467 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-11T14:41:22.467 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 111, 2024-09-11T14:41:22.467 INFO:teuthology.orchestra.run.smithi181.stdout: "pg": "3.e3b9a217", 2024-09-11T14:41:22.468 INFO:teuthology.orchestra.run.smithi181.stdout: "osd": 3, 2024-09-11T14:41:22.468 INFO:teuthology.orchestra.run.smithi181.stdout: "object_id": "benchmark_data_smithi181_30163_object110", 2024-09-11T14:41:22.468 INFO:teuthology.orchestra.run.smithi181.stdout: "object_locator": "@3", 2024-09-11T14:41:22.468 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_id": "benchmark_data_smithi181_30163_object110", 2024-09-11T14:41:22.468 INFO:teuthology.orchestra.run.smithi181.stdout: "target_object_locator": "@3", 2024-09-11T14:41:22.468 INFO:teuthology.orchestra.run.smithi181.stdout: "paused": 0, 2024-09-11T14:41:22.468 INFO:teuthology.orchestra.run.smithi181.stdout: "used_replica": 0, 2024-09-11T14:41:22.468 INFO:teuthology.orchestra.run.smithi181.stdout: "precalc_pgid": 0, 2024-09-11T14:41:22.468 INFO:teuthology.orchestra.run.smithi181.stdout: "last_sent": "485.209102s", 2024-09-11T14:41:22.468 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0099998039999999993, 2024-09-11T14:41:22.468 INFO:teuthology.orchestra.run.smithi181.stdout: "attempts": 1, 2024-09-11T14:41:22.468 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": "head", 2024-09-11T14:41:22.468 INFO:teuthology.orchestra.run.smithi181.stdout: "snap_context": "0=[]", 2024-09-11T14:41:22.469 INFO:teuthology.orchestra.run.smithi181.stdout: "mtime": "2024-09-11T14:41:22.431664+0000", 2024-09-11T14:41:22.469 INFO:teuthology.orchestra.run.smithi181.stdout: "osd_ops": [ 2024-09-11T14:41:22.469 INFO:teuthology.orchestra.run.smithi181.stdout: "set-alloc-hint object_size 65536 write_size 65536", 2024-09-11T14:41:22.469 INFO:teuthology.orchestra.run.smithi181.stdout: "write 0~65536 in=65536b" 2024-09-11T14:41:22.469 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-11T14:41:22.469 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-11T14:41:22.469 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-11T14:41:22.469 INFO:teuthology.orchestra.run.smithi181.stdout: "linger_ops": [], 2024-09-11T14:41:22.469 INFO:teuthology.orchestra.run.smithi181.stdout: "pool_ops": [], 2024-09-11T14:41:22.469 INFO:teuthology.orchestra.run.smithi181.stdout: "pool_stat_ops": [], 2024-09-11T14:41:22.469 INFO:teuthology.orchestra.run.smithi181.stdout: "statfs_ops": [], 2024-09-11T14:41:22.469 INFO:teuthology.orchestra.run.smithi181.stdout: "command_ops": [] 2024-09-11T14:41:22.469 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T14:41:22.470 DEBUG:tasks.admin_socket:admin socket command objecter_requests returned { "ops": [ { "tid": 80, "pg": "3.3adab0dc", "osd": 0, "object_id": "benchmark_data_smithi181_30163_object79", "object_locator": "@3", "target_object_id": "benchmark_data_smithi181_30163_object79", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "484.666112s", "age": 0.552989179, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-11T14:41:21.888650+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 83, "pg": "3.a9f591bc", "osd": 0, "object_id": "benchmark_data_smithi181_30163_object82", "object_locator": "@3", "target_object_id": "benchmark_data_smithi181_30163_object82", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "484.698112s", "age": 0.52098980500000003, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-11T14:41:21.920250+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 91, "pg": "3.b53b711f", "osd": 0, "object_id": "benchmark_data_smithi181_30163_object90", "object_locator": "@3", "target_object_id": "benchmark_data_smithi181_30163_object90", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "484.901108s", "age": 0.317993778, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-11T14:41:22.123362+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 107, "pg": "3.5b0c460c", "osd": 0, "object_id": "benchmark_data_smithi181_30163_object106", "object_locator": "@3", "target_object_id": "benchmark_data_smithi181_30163_object106", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "485.126103s", "age": 0.09299818, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-11T14:41:22.348070+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 110, "pg": "3.1f868bbc", "osd": 0, "object_id": "benchmark_data_smithi181_30163_object109", "object_locator": "@3", "target_object_id": "benchmark_data_smithi181_30163_object109", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "485.191102s", "age": 0.027999452000000001, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-11T14:41:22.413583+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 76, "pg": "3.89fcc6f2", "osd": 1, "object_id": "benchmark_data_smithi181_30163_object75", "object_locator": "@3", "target_object_id": "benchmark_data_smithi181_30163_object75", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "484.575114s", "age": 0.64398739900000002, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-11T14:41:21.797504+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 93, "pg": "3.8dd1fb5d", "osd": 1, "object_id": "benchmark_data_smithi181_30163_object92", "object_locator": "@3", "target_object_id": "benchmark_data_smithi181_30163_object92", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "484.938107s", "age": 0.28099450199999998, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-11T14:41:22.160732+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 94, "pg": "3.6da523d5", "osd": 1, "object_id": "benchmark_data_smithi181_30163_object93", "object_locator": "@3", "target_object_id": "benchmark_data_smithi181_30163_object93", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "484.944107s", "age": 0.274994619, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-11T14:41:22.166456+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 96, "pg": "3.5062bc82", "osd": 1, "object_id": "benchmark_data_smithi181_30163_object95", "object_locator": "@3", "target_object_id": "benchmark_data_smithi181_30163_object95", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "484.987106s", "age": 0.23199545999999999, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-11T14:41:22.209644+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 101, "pg": "3.ce43f5c5", "osd": 1, "object_id": "benchmark_data_smithi181_30163_object100", "object_locator": "@3", "target_object_id": "benchmark_data_smithi181_30163_object100", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "485.074104s", "age": 0.14499716300000001, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-11T14:41:22.296638+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 103, "pg": "3.1120432d", "osd": 1, "object_id": "benchmark_data_smithi181_30163_object102", "object_locator": "@3", "target_object_id": "benchmark_data_smithi181_30163_object102", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "485.104104s", "age": 0.11499775, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-11T14:41:22.325985+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 108, "pg": "3.baab5292", "osd": 1, "object_id": "benchmark_data_smithi181_30163_object107", "object_locator": "@3", "target_object_id": "benchmark_data_smithi181_30163_object107", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "485.162103s", "age": 0.056998884999999999, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-11T14:41:22.384818+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 105, "pg": "3.a6e91216", "osd": 2, "object_id": "benchmark_data_smithi181_30163_object104", "object_locator": "@3", "target_object_id": "benchmark_data_smithi181_30163_object104", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "485.114104s", "age": 0.104997945, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-11T14:41:22.336182+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 109, "pg": "3.ab15e626", "osd": 2, "object_id": "benchmark_data_smithi181_30163_object108", "object_locator": "@3", "target_object_id": "benchmark_data_smithi181_30163_object108", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "485.178102s", "age": 0.040999198000000001, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-11T14:41:22.400323+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 106, "pg": "3.d1b3a230", "osd": 3, "object_id": "benchmark_data_smithi181_30163_object105", "object_locator": "@3", "target_object_id": "benchmark_data_smithi181_30163_object105", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "485.114104s", "age": 0.104997945, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-11T14:41:22.336226+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] }, { "tid": 111, "pg": "3.e3b9a217", "osd": 3, "object_id": "benchmark_data_smithi181_30163_object110", "object_locator": "@3", "target_object_id": "benchmark_data_smithi181_30163_object110", "target_object_locator": "@3", "paused": 0, "used_replica": 0, "precalc_pgid": 0, "last_sent": "485.209102s", "age": 0.0099998039999999993, "attempts": 1, "snapid": "head", "snap_context": "0=[]", "mtime": "2024-09-11T14:41:22.431664+0000", "osd_ops": [ "set-alloc-hint object_size 65536 write_size 65536", "write 0~65536 in=65536b" ] } ], "linger_ops": [], "pool_ops": [], "pool_stat_ops": [], "statfs_ops": [], "command_ops": [] } 2024-09-11T14:41:22.471 DEBUG:teuthology.orchestra.run.smithi181:> /home/ubuntu/cephtest/admin_socket_client.0/objecter_requests 2024-09-11T14:41:22.501 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:22.497+0000 7fc2e087f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:22.508 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:22.507+0000 7fc2e087f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:22.515 DEBUG:teuthology.orchestra.run.smithi181:> rm -rf -- /home/ubuntu/cephtest/admin_socket_client.0 2024-09-11T14:41:22.572 DEBUG:teuthology.parallel:result is None 2024-09-11T14:41:22.572 DEBUG:teuthology.run_tasks:Unwinding manager radosbench 2024-09-11T14:41:22.581 INFO:tasks.radosbench:joining radosbench (timing out after 4800s) 2024-09-11T14:41:22.581 INFO:teuthology.orchestra.run:waiting for 4800 2024-09-11T14:41:22.903 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 2 16 143 127 3.96847 3.8125 0.0669328 0.22017 2024-09-11T14:41:23.481 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:41:23.482 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:41:23.482 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-11T14:41:23.482 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-11T14:41:23.641 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:23.629+0000 7fc28b791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:23.647 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:23.645+0000 7fc28b791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:23.718 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:41:23.718 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:41:23.718 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:41:23.718 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:41:23.718 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:41:23.719 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:41:23.719 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:41:23.719 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:41:23.719 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:41:23.719 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:41:23.719 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:41:23.719 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:41:23.721 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:41:23.721 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:41:23.722 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-11T14:41:23.881 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:23.869+0000 7f7a7c191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:23.886 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:23.885+0000 7f7a7c191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:23.903 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 3 16 220 204 4.24969 4.8125 0.176249 0.222264 2024-09-11T14:41:23.935 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:41:23.935 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:41:23.935 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:41:23.935 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:41:23.935 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:41:23.935 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:41:23.936 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:41:23.936 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:41:23.936 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:41:23.936 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:41:23.936 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:41:23.936 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:41:23.938 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:41:23.939 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:41:23.939 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-11T14:41:24.082 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:24.074+0000 7f6afd0c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:24.087 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:24.086+0000 7f6afd0c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:24.113 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:41:24.113 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:41:24.113 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:41:24.113 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:41:24.113 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:41:24.113 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:41:24.113 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:41:24.113 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:41:24.113 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:41:24.113 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:41:24.113 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:41:24.113 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:41:24.116 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:41:24.116 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:41:24.116 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-11T14:41:24.269 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:24.261+0000 7f584bed1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:24.274 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:24.272+0000 7f584bed1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:24.322 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:41:24.323 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:41:24.323 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:41:24.323 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:41:24.323 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:41:24.323 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:41:24.323 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:41:24.323 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:41:24.323 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:41:24.323 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:41:24.323 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:41:24.324 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:41:24.326 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:41:24.326 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:41:24.763 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:41:24.903 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 4 16 297 281 4.39028 4.8125 0.230422 0.22016 2024-09-11T14:41:24.907 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:24.902+0000 7f8ccaf91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:24.913 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:24.912+0000 7f8ccaf91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:25.903 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 5 16 374 358 4.47462 4.8125 0.038316 0.214609 2024-09-11T14:41:26.903 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 6 16 457 441 4.59336 5.1875 0.110338 0.21132 2024-09-11T14:41:27.229 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:41:27.370 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:27.365+0000 7f4857151640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:27.375 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:27.373+0000 7f4857151640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:27.904 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 7 16 544 528 4.71386 5.4375 0.0636353 0.207716 2024-09-11T14:41:28.904 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 8 16 632 616 4.81205 5.5 0.0556346 0.20473 2024-09-11T14:41:29.327 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:41:29.327 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-11T14:41:29.487 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:29.477+0000 7f6fb9111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:29.493 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:29.492+0000 7f6fb9111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:29.752 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:41:29.898 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:29.894+0000 7f580a948640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:29.903 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:29.902+0000 7f580a948640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:29.903 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 9 16 729 713 4.95093 6.0625 0.275564 0.199295 2024-09-11T14:41:30.834 INFO:teuthology.orchestra.run.smithi181.stderr:reweighted osd.0 to 1 (10000) 2024-09-11T14:41:30.853 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-11T14:41:30.904 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 10 16 820 804 5.02453 5.6875 0.06371 0.195859 2024-09-11T14:41:31.011 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:30.999+0000 7fc504aa3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:31.015 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:31.014+0000 7fc504aa3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:31.835 INFO:teuthology.orchestra.run.smithi181.stderr:reweighted osd.1 to 1 (10000) 2024-09-11T14:41:31.861 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-11T14:41:31.904 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 11 16 912 896 5.09045 5.75 0.196708 0.193643 2024-09-11T14:41:32.002 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:31.992+0000 7fa85ef91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:32.009 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:32.007+0000 7fa85ef91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:32.210 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:41:32.363 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:32.359+0000 7f04f5cd1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:32.367 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:32.366+0000 7f04f5cd1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:32.842 INFO:teuthology.orchestra.run.smithi181.stderr:reweighted osd.2 to 1 (10000) 2024-09-11T14:41:32.865 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-11T14:41:32.904 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 12 16 995 979 5.09849 5.1875 0.177497 0.193839 2024-09-11T14:41:33.011 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:33.000+0000 7fb898a74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:33.016 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:33.015+0000 7fb898a74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:33.843 INFO:teuthology.orchestra.run.smithi181.stderr:reweighted osd.3 to 1 (10000) 2024-09-11T14:41:33.871 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-11T14:41:33.871 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:41:33.904 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 13 16 1090 1074 5.16299 5.9375 0.0590902 0.191955 2024-09-11T14:41:34.066 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:34.057+0000 7f158306c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:34.075 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:34.074+0000 7f158306c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:34.444 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:41:34.445 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:41:34.471 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-09-11T14:41:33.847647+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":49414707,"num_objects":749,"num_object_clones":0,"num_object_copies":2247,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":826,"num_write_kb":48788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":800,"ondisk_log_size":800,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":255804,"kb_used_data":255804,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":208935108,"statfs":{"total":214211493888,"available":213949550592,"internally_reserved":0,"allocated":261943296,"data_stored":261943296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28246016,"num_objects":431,"num_object_clones":0,"num_object_copies":1293,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":431,"num_write_kb":27584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.138098"},"pg_stats":[{"pgid":"3.f","version":"19'36","reported_seq":48,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.412057+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:41:29.412057+0000","last_peered":"2024-09-11T14:41:29.412057+0000","last_clean":"2024-09-11T14:41:29.412057+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:41:29.412057+0000","last_undegraded":"2024-09-11T14:41:29.412057+0000","last_fullsized":"2024-09-11T14:41:29.412057+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2359296,"num_objects":36,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36,"num_write_kb":2304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"19'68","reported_seq":80,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:30.525830+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:41:30.525830+0000","last_peered":"2024-09-11T14:41:30.525830+0000","last_clean":"2024-09-11T14:41:30.525830+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:41:30.525830+0000","last_undegraded":"2024-09-11T14:41:30.525830+0000","last_fullsized":"2024-09-11T14:41:30.525830+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4456448,"num_objects":68,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68,"num_write_kb":4352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"19'35","reported_seq":47,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.709203+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:41:29.709203+0000","last_peered":"2024-09-11T14:41:29.709203+0000","last_clean":"2024-09-11T14:41:29.709203+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:41:29.709203+0000","last_undegraded":"2024-09-11T14:41:29.709203+0000","last_fullsized":"2024-09-11T14:41:29.709203+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2293760,"num_objects":35,"num_object_clones":0,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35,"num_write_kb":2240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"19'54","reported_seq":66,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.597772+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:41:29.597772+0000","last_peered":"2024-09-11T14:41:29.597772+0000","last_clean":"2024-09-11T14:41:29.597772+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:41:29.597772+0000","last_undegraded":"2024-09-11T14:41:29.597772+0000","last_fullsized":"2024-09-11T14:41:29.597772+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3538944,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54,"num_write_kb":3456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"19'46","reported_seq":58,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.432279+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:41:29.432279+0000","last_peered":"2024-09-11T14:41:29.432279+0000","last_clean":"2024-09-11T14:41:29.432279+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:41:29.432279+0000","last_undegraded":"2024-09-11T14:41:29.432279+0000","last_fullsized":"2024-09-11T14:41:29.432279+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3014656,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":2944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"19'48","reported_seq":60,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.748491+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:41:29.748491+0000","last_peered":"2024-09-11T14:41:29.748491+0000","last_clean":"2024-09-11T14:41:29.748491+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:41:29.748491+0000","last_undegraded":"2024-09-11T14:41:29.748491+0000","last_fullsized":"2024-09-11T14:41:29.748491+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"19'33","reported_seq":45,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.108110+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:41:29.108110+0000","last_peered":"2024-09-11T14:41:29.108110+0000","last_clean":"2024-09-11T14:41:29.108110+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:41:29.108110+0000","last_undegraded":"2024-09-11T14:41:29.108110+0000","last_fullsized":"2024-09-11T14:41:29.108110+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2162688,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":2112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"19'41","reported_seq":53,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.141380+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:41:29.141380+0000","last_peered":"2024-09-11T14:41:29.141380+0000","last_clean":"2024-09-11T14:41:29.141380+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:41:29.141380+0000","last_undegraded":"2024-09-11T14:41:29.141380+0000","last_fullsized":"2024-09-11T14:41:29.141380+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2686976,"num_objects":41,"num_object_clones":0,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41,"num_write_kb":2624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":97,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:20.852495+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:41:20.852495+0000","last_peered":"2024-09-11T14:41:20.852495+0000","last_clean":"2024-09-11T14:41:20.852495+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:41:20.852495+0000","last_undegraded":"2024-09-11T14:41:20.852495+0000","last_fullsized":"2024-09-11T14:41:20.852495+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:20.858684+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:41:20.858684+0000","last_peered":"2024-09-11T14:41:20.858684+0000","last_clean":"2024-09-11T14:41:20.858684+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:41:20.858684+0000","last_undegraded":"2024-09-11T14:41:20.858684+0000","last_fullsized":"2024-09-11T14:41:20.858684+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"19'42","reported_seq":54,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.590263+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:41:29.590263+0000","last_peered":"2024-09-11T14:41:29.590263+0000","last_clean":"2024-09-11T14:41:29.590263+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:41:29.590263+0000","last_undegraded":"2024-09-11T14:41:29.590263+0000","last_fullsized":"2024-09-11T14:41:29.590263+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2752512,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42,"num_write_kb":2688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":25,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:20.854776+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:41:20.854776+0000","last_peered":"2024-09-11T14:41:20.854776+0000","last_clean":"2024-09-11T14:41:20.854776+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:41:20.854776+0000","last_undegraded":"2024-09-11T14:41:20.854776+0000","last_fullsized":"2024-09-11T14:41:20.854776+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"19'41","reported_seq":53,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.353129+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:41:29.353129+0000","last_peered":"2024-09-11T14:41:29.353129+0000","last_clean":"2024-09-11T14:41:29.353129+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:41:29.353129+0000","last_undegraded":"2024-09-11T14:41:29.353129+0000","last_fullsized":"2024-09-11T14:41:29.353129+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2686976,"num_objects":41,"num_object_clones":0,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41,"num_write_kb":2624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:20.851661+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:41:20.851661+0000","last_peered":"2024-09-11T14:41:20.851661+0000","last_clean":"2024-09-11T14:41:20.851661+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:41:20.851661+0000","last_undegraded":"2024-09-11T14:41:20.851661+0000","last_fullsized":"2024-09-11T14:41:20.851661+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"19'56","reported_seq":68,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.523300+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:41:29.523300+0000","last_peered":"2024-09-11T14:41:29.523300+0000","last_clean":"2024-09-11T14:41:29.523300+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:41:29.523300+0000","last_undegraded":"2024-09-11T14:41:29.523300+0000","last_fullsized":"2024-09-11T14:41:29.523300+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3670016,"num_objects":56,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56,"num_write_kb":3584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:20.855755+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:41:20.855755+0000","last_peered":"2024-09-11T14:41:20.855755+0000","last_clean":"2024-09-11T14:41:20.855755+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:41:20.855755+0000","last_undegraded":"2024-09-11T14:41:20.855755+0000","last_fullsized":"2024-09-11T14:41:20.855755+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"19'47","reported_seq":59,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:30.291721+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:41:30.291721+0000","last_peered":"2024-09-11T14:41:30.291721+0000","last_clean":"2024-09-11T14:41:30.291721+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:41:30.291721+0000","last_undegraded":"2024-09-11T14:41:30.291721+0000","last_fullsized":"2024-09-11T14:41:30.291721+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3080192,"num_objects":47,"num_object_clones":0,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":3008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"19'45","reported_seq":57,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.687569+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:41:29.687569+0000","last_peered":"2024-09-11T14:41:29.687569+0000","last_clean":"2024-09-11T14:41:29.687569+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:41:29.687569+0000","last_undegraded":"2024-09-11T14:41:29.687569+0000","last_fullsized":"2024-09-11T14:41:29.687569+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2949120,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":2880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:20.855868+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:41:20.855868+0000","last_peered":"2024-09-11T14:41:20.855868+0000","last_clean":"2024-09-11T14:41:20.855868+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:41:20.855868+0000","last_undegraded":"2024-09-11T14:41:20.855868+0000","last_fullsized":"2024-09-11T14:41:20.855868+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"19'42","reported_seq":54,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:30.533620+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:41:30.533620+0000","last_peered":"2024-09-11T14:41:30.533620+0000","last_clean":"2024-09-11T14:41:30.533620+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:41:30.533620+0000","last_undegraded":"2024-09-11T14:41:30.533620+0000","last_fullsized":"2024-09-11T14:41:30.533620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2752512,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42,"num_write_kb":2688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:20.850469+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:41:20.850469+0000","last_peered":"2024-09-11T14:41:20.850469+0000","last_clean":"2024-09-11T14:41:20.850469+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:41:20.850469+0000","last_undegraded":"2024-09-11T14:41:20.850469+0000","last_fullsized":"2024-09-11T14:41:20.850469+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"19'51","reported_seq":63,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.573151+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:41:29.573151+0000","last_peered":"2024-09-11T14:41:29.573151+0000","last_clean":"2024-09-11T14:41:29.573151+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:41:29.573151+0000","last_undegraded":"2024-09-11T14:41:29.573151+0000","last_fullsized":"2024-09-11T14:41:29.573151+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3342336,"num_objects":51,"num_object_clones":0,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51,"num_write_kb":3264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":25,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:20.856195+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:41:20.856195+0000","last_peered":"2024-09-11T14:41:20.856195+0000","last_clean":"2024-09-11T14:41:20.856195+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:41:20.856195+0000","last_undegraded":"2024-09-11T14:41:20.856195+0000","last_fullsized":"2024-09-11T14:41:20.856195+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"19'60","reported_seq":72,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:30.692828+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:41:30.692828+0000","last_peered":"2024-09-11T14:41:30.692828+0000","last_clean":"2024-09-11T14:41:30.692828+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:41:30.692828+0000","last_undegraded":"2024-09-11T14:41:30.692828+0000","last_fullsized":"2024-09-11T14:41:30.692828+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3932160,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60,"num_write_kb":3840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:20.858443+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:41:20.858443+0000","last_peered":"2024-09-11T14:41:20.858443+0000","last_clean":"2024-09-11T14:41:20.858443+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:41:20.858443+0000","last_undegraded":"2024-09-11T14:41:20.858443+0000","last_fullsized":"2024-09-11T14:41:20.858443+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":48824320,"num_objects":745,"num_object_clones":0,"num_object_copies":2235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":745,"num_write_kb":47680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":213911605248,"internally_reserved":0,"allocated":299888640,"data_stored":299888640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":745,"ondisk_log_size":745,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":213911736320,"internally_reserved":0,"allocated":299757568,"data_stored":299757568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53467095040,"internally_reserved":0,"allocated":85778432,"data_stored":85778432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":68908,"kb_used_data":68908,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52228820,"statfs":{"total":53552873472,"available":53482311680,"internally_reserved":0,"allocated":70561792,"data_stored":70561792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705688,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":72848,"kb_used_data":72848,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52224880,"statfs":{"total":53552873472,"available":53478277120,"internally_reserved":0,"allocated":74596352,"data_stored":74596352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705688,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":59908,"kb_used_data":59908,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52237820,"statfs":{"total":53552873472,"available":53491527680,"internally_reserved":0,"allocated":61345792,"data_stored":61345792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705688,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":54140,"kb_used_data":54140,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52243588,"statfs":{"total":53552873472,"available":53497434112,"internally_reserved":0,"allocated":55439360,"data_stored":55439360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53467095040,"internally_reserved":0,"allocated":85778432,"data_stored":85778432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53490012160,"internally_reserved":0,"allocated":62861312,"data_stored":62861312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53480869888,"internally_reserved":0,"allocated":72003584,"data_stored":72003584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53473759232,"internally_reserved":0,"allocated":79114240,"data_stored":79114240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53467095040,"internally_reserved":0,"allocated":85778432,"data_stored":85778432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":53490012160,"internally_reserved":0,"allocated":62861312,"data_stored":62861312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":53480869888,"internally_reserved":0,"allocated":72003584,"data_stored":72003584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":53473628160,"internally_reserved":0,"allocated":79245312,"data_stored":79245312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":53467095040,"internally_reserved":0,"allocated":85778432,"data_stored":85778432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:41:34.472 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:41:34.653 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:34.643+0000 7fdaa2911640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:34.659 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:34.658+0000 7fdaa2911640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:34.718 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:41:34.858 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:34.854+0000 7f047e591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:34.865 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:34.864+0000 7f047e591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:34.904 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 14 16 1185 1169 5.21828 5.9375 0.0552488 0.189956 2024-09-11T14:41:35.032 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:41:35.033 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:41:35.054 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-09-11T14:41:34.710743+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":49414707,"num_objects":749,"num_object_clones":0,"num_object_copies":2247,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":826,"num_write_kb":48788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":800,"ondisk_log_size":800,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":255804,"kb_used_data":255804,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":208935108,"statfs":{"total":214211493888,"available":213949550592,"internally_reserved":0,"allocated":261943296,"data_stored":261943296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7405568,"num_objects":113,"num_object_clones":0,"num_object_copies":339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":113,"num_write_kb":7232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000734"},"pg_stats":[{"pgid":"3.f","version":"19'36","reported_seq":48,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.412057+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:41:29.412057+0000","last_peered":"2024-09-11T14:41:29.412057+0000","last_clean":"2024-09-11T14:41:29.412057+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:41:29.412057+0000","last_undegraded":"2024-09-11T14:41:29.412057+0000","last_fullsized":"2024-09-11T14:41:29.412057+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2359296,"num_objects":36,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36,"num_write_kb":2304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"19'68","reported_seq":80,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:30.525830+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:41:30.525830+0000","last_peered":"2024-09-11T14:41:30.525830+0000","last_clean":"2024-09-11T14:41:30.525830+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:41:30.525830+0000","last_undegraded":"2024-09-11T14:41:30.525830+0000","last_fullsized":"2024-09-11T14:41:30.525830+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4456448,"num_objects":68,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68,"num_write_kb":4352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"19'35","reported_seq":47,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.709203+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:41:29.709203+0000","last_peered":"2024-09-11T14:41:29.709203+0000","last_clean":"2024-09-11T14:41:29.709203+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:41:29.709203+0000","last_undegraded":"2024-09-11T14:41:29.709203+0000","last_fullsized":"2024-09-11T14:41:29.709203+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2293760,"num_objects":35,"num_object_clones":0,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35,"num_write_kb":2240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"19'54","reported_seq":66,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.597772+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:41:29.597772+0000","last_peered":"2024-09-11T14:41:29.597772+0000","last_clean":"2024-09-11T14:41:29.597772+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:41:29.597772+0000","last_undegraded":"2024-09-11T14:41:29.597772+0000","last_fullsized":"2024-09-11T14:41:29.597772+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3538944,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54,"num_write_kb":3456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"19'46","reported_seq":58,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.432279+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:41:29.432279+0000","last_peered":"2024-09-11T14:41:29.432279+0000","last_clean":"2024-09-11T14:41:29.432279+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:41:29.432279+0000","last_undegraded":"2024-09-11T14:41:29.432279+0000","last_fullsized":"2024-09-11T14:41:29.432279+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3014656,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":2944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"19'48","reported_seq":60,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.748491+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:41:29.748491+0000","last_peered":"2024-09-11T14:41:29.748491+0000","last_clean":"2024-09-11T14:41:29.748491+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:41:29.748491+0000","last_undegraded":"2024-09-11T14:41:29.748491+0000","last_fullsized":"2024-09-11T14:41:29.748491+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"19'33","reported_seq":45,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.108110+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:41:29.108110+0000","last_peered":"2024-09-11T14:41:29.108110+0000","last_clean":"2024-09-11T14:41:29.108110+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:41:29.108110+0000","last_undegraded":"2024-09-11T14:41:29.108110+0000","last_fullsized":"2024-09-11T14:41:29.108110+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2162688,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":2112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"19'41","reported_seq":53,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.141380+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:41:29.141380+0000","last_peered":"2024-09-11T14:41:29.141380+0000","last_clean":"2024-09-11T14:41:29.141380+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:41:29.141380+0000","last_undegraded":"2024-09-11T14:41:29.141380+0000","last_fullsized":"2024-09-11T14:41:29.141380+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2686976,"num_objects":41,"num_object_clones":0,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41,"num_write_kb":2624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":97,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:20.852495+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:41:20.852495+0000","last_peered":"2024-09-11T14:41:20.852495+0000","last_clean":"2024-09-11T14:41:20.852495+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:41:20.852495+0000","last_undegraded":"2024-09-11T14:41:20.852495+0000","last_fullsized":"2024-09-11T14:41:20.852495+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:20.858684+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:41:20.858684+0000","last_peered":"2024-09-11T14:41:20.858684+0000","last_clean":"2024-09-11T14:41:20.858684+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:41:20.858684+0000","last_undegraded":"2024-09-11T14:41:20.858684+0000","last_fullsized":"2024-09-11T14:41:20.858684+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"19'42","reported_seq":54,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.590263+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:41:29.590263+0000","last_peered":"2024-09-11T14:41:29.590263+0000","last_clean":"2024-09-11T14:41:29.590263+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:41:29.590263+0000","last_undegraded":"2024-09-11T14:41:29.590263+0000","last_fullsized":"2024-09-11T14:41:29.590263+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2752512,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42,"num_write_kb":2688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":25,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:20.854776+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:41:20.854776+0000","last_peered":"2024-09-11T14:41:20.854776+0000","last_clean":"2024-09-11T14:41:20.854776+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:41:20.854776+0000","last_undegraded":"2024-09-11T14:41:20.854776+0000","last_fullsized":"2024-09-11T14:41:20.854776+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"19'41","reported_seq":53,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.353129+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:41:29.353129+0000","last_peered":"2024-09-11T14:41:29.353129+0000","last_clean":"2024-09-11T14:41:29.353129+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:41:29.353129+0000","last_undegraded":"2024-09-11T14:41:29.353129+0000","last_fullsized":"2024-09-11T14:41:29.353129+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2686976,"num_objects":41,"num_object_clones":0,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41,"num_write_kb":2624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:20.851661+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:41:20.851661+0000","last_peered":"2024-09-11T14:41:20.851661+0000","last_clean":"2024-09-11T14:41:20.851661+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:41:20.851661+0000","last_undegraded":"2024-09-11T14:41:20.851661+0000","last_fullsized":"2024-09-11T14:41:20.851661+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"19'56","reported_seq":68,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.523300+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:41:29.523300+0000","last_peered":"2024-09-11T14:41:29.523300+0000","last_clean":"2024-09-11T14:41:29.523300+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:41:29.523300+0000","last_undegraded":"2024-09-11T14:41:29.523300+0000","last_fullsized":"2024-09-11T14:41:29.523300+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3670016,"num_objects":56,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56,"num_write_kb":3584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:20.855755+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:41:20.855755+0000","last_peered":"2024-09-11T14:41:20.855755+0000","last_clean":"2024-09-11T14:41:20.855755+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:41:20.855755+0000","last_undegraded":"2024-09-11T14:41:20.855755+0000","last_fullsized":"2024-09-11T14:41:20.855755+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"19'47","reported_seq":59,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:30.291721+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:41:30.291721+0000","last_peered":"2024-09-11T14:41:30.291721+0000","last_clean":"2024-09-11T14:41:30.291721+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:41:30.291721+0000","last_undegraded":"2024-09-11T14:41:30.291721+0000","last_fullsized":"2024-09-11T14:41:30.291721+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3080192,"num_objects":47,"num_object_clones":0,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":3008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"19'45","reported_seq":57,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.687569+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:41:29.687569+0000","last_peered":"2024-09-11T14:41:29.687569+0000","last_clean":"2024-09-11T14:41:29.687569+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:41:29.687569+0000","last_undegraded":"2024-09-11T14:41:29.687569+0000","last_fullsized":"2024-09-11T14:41:29.687569+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2949120,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":2880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:20.855868+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:41:20.855868+0000","last_peered":"2024-09-11T14:41:20.855868+0000","last_clean":"2024-09-11T14:41:20.855868+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:41:20.855868+0000","last_undegraded":"2024-09-11T14:41:20.855868+0000","last_fullsized":"2024-09-11T14:41:20.855868+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"19'42","reported_seq":54,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:30.533620+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:41:30.533620+0000","last_peered":"2024-09-11T14:41:30.533620+0000","last_clean":"2024-09-11T14:41:30.533620+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:41:30.533620+0000","last_undegraded":"2024-09-11T14:41:30.533620+0000","last_fullsized":"2024-09-11T14:41:30.533620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2752512,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42,"num_write_kb":2688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:20.850469+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:41:20.850469+0000","last_peered":"2024-09-11T14:41:20.850469+0000","last_clean":"2024-09-11T14:41:20.850469+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:41:20.850469+0000","last_undegraded":"2024-09-11T14:41:20.850469+0000","last_fullsized":"2024-09-11T14:41:20.850469+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"19'51","reported_seq":63,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.573151+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:41:29.573151+0000","last_peered":"2024-09-11T14:41:29.573151+0000","last_clean":"2024-09-11T14:41:29.573151+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:41:29.573151+0000","last_undegraded":"2024-09-11T14:41:29.573151+0000","last_fullsized":"2024-09-11T14:41:29.573151+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3342336,"num_objects":51,"num_object_clones":0,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51,"num_write_kb":3264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":25,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:20.856195+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:41:20.856195+0000","last_peered":"2024-09-11T14:41:20.856195+0000","last_clean":"2024-09-11T14:41:20.856195+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:41:20.856195+0000","last_undegraded":"2024-09-11T14:41:20.856195+0000","last_fullsized":"2024-09-11T14:41:20.856195+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"19'60","reported_seq":72,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:30.692828+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:41:30.692828+0000","last_peered":"2024-09-11T14:41:30.692828+0000","last_clean":"2024-09-11T14:41:30.692828+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:41:30.692828+0000","last_undegraded":"2024-09-11T14:41:30.692828+0000","last_fullsized":"2024-09-11T14:41:30.692828+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3932160,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60,"num_write_kb":3840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:20.858443+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:41:20.858443+0000","last_peered":"2024-09-11T14:41:20.858443+0000","last_clean":"2024-09-11T14:41:20.858443+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:41:20.858443+0000","last_undegraded":"2024-09-11T14:41:20.858443+0000","last_fullsized":"2024-09-11T14:41:20.858443+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":48824320,"num_objects":745,"num_object_clones":0,"num_object_copies":2235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":745,"num_write_kb":47680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":213911605248,"internally_reserved":0,"allocated":299888640,"data_stored":299888640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":745,"ondisk_log_size":745,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":213911736320,"internally_reserved":0,"allocated":299757568,"data_stored":299757568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53467095040,"internally_reserved":0,"allocated":85778432,"data_stored":85778432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":68908,"kb_used_data":68908,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52228820,"statfs":{"total":53552873472,"available":53482311680,"internally_reserved":0,"allocated":70561792,"data_stored":70561792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705688,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":72848,"kb_used_data":72848,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52224880,"statfs":{"total":53552873472,"available":53478277120,"internally_reserved":0,"allocated":74596352,"data_stored":74596352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705688,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":59908,"kb_used_data":59908,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52237820,"statfs":{"total":53552873472,"available":53491527680,"internally_reserved":0,"allocated":61345792,"data_stored":61345792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705688,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":54140,"kb_used_data":54140,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52243588,"statfs":{"total":53552873472,"available":53497434112,"internally_reserved":0,"allocated":55439360,"data_stored":55439360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53467095040,"internally_reserved":0,"allocated":85778432,"data_stored":85778432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53490012160,"internally_reserved":0,"allocated":62861312,"data_stored":62861312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53480869888,"internally_reserved":0,"allocated":72003584,"data_stored":72003584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53473759232,"internally_reserved":0,"allocated":79114240,"data_stored":79114240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53467095040,"internally_reserved":0,"allocated":85778432,"data_stored":85778432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":53490012160,"internally_reserved":0,"allocated":62861312,"data_stored":62861312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":53480869888,"internally_reserved":0,"allocated":72003584,"data_stored":72003584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":53473628160,"internally_reserved":0,"allocated":79245312,"data_stored":79245312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":53467095040,"internally_reserved":0,"allocated":85778432,"data_stored":85778432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:41:35.055 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-11T14:41:35.055 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:41:35.056 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:41:35.190 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:35.181+0000 7f2e7b391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:35.198 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:35.197+0000 7f2e7b391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:35.541 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:41:35.541 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:41:35.565 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-09-11T14:41:34.710743+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":49414707,"num_objects":749,"num_object_clones":0,"num_object_copies":2247,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":826,"num_write_kb":48788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":800,"ondisk_log_size":800,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":255804,"kb_used_data":255804,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":208935108,"statfs":{"total":214211493888,"available":213949550592,"internally_reserved":0,"allocated":261943296,"data_stored":261943296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7405568,"num_objects":113,"num_object_clones":0,"num_object_copies":339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":113,"num_write_kb":7232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000734"},"pg_stats":[{"pgid":"3.f","version":"19'36","reported_seq":48,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.412057+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:41:29.412057+0000","last_peered":"2024-09-11T14:41:29.412057+0000","last_clean":"2024-09-11T14:41:29.412057+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:41:29.412057+0000","last_undegraded":"2024-09-11T14:41:29.412057+0000","last_fullsized":"2024-09-11T14:41:29.412057+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2359296,"num_objects":36,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36,"num_write_kb":2304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"19'68","reported_seq":80,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:30.525830+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:41:30.525830+0000","last_peered":"2024-09-11T14:41:30.525830+0000","last_clean":"2024-09-11T14:41:30.525830+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:41:30.525830+0000","last_undegraded":"2024-09-11T14:41:30.525830+0000","last_fullsized":"2024-09-11T14:41:30.525830+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4456448,"num_objects":68,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68,"num_write_kb":4352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"19'35","reported_seq":47,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.709203+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:41:29.709203+0000","last_peered":"2024-09-11T14:41:29.709203+0000","last_clean":"2024-09-11T14:41:29.709203+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:41:29.709203+0000","last_undegraded":"2024-09-11T14:41:29.709203+0000","last_fullsized":"2024-09-11T14:41:29.709203+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2293760,"num_objects":35,"num_object_clones":0,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35,"num_write_kb":2240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"19'54","reported_seq":66,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.597772+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:41:29.597772+0000","last_peered":"2024-09-11T14:41:29.597772+0000","last_clean":"2024-09-11T14:41:29.597772+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:41:29.597772+0000","last_undegraded":"2024-09-11T14:41:29.597772+0000","last_fullsized":"2024-09-11T14:41:29.597772+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3538944,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54,"num_write_kb":3456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"19'46","reported_seq":58,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.432279+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:41:29.432279+0000","last_peered":"2024-09-11T14:41:29.432279+0000","last_clean":"2024-09-11T14:41:29.432279+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:41:29.432279+0000","last_undegraded":"2024-09-11T14:41:29.432279+0000","last_fullsized":"2024-09-11T14:41:29.432279+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3014656,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":2944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"19'48","reported_seq":60,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.748491+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:41:29.748491+0000","last_peered":"2024-09-11T14:41:29.748491+0000","last_clean":"2024-09-11T14:41:29.748491+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:41:29.748491+0000","last_undegraded":"2024-09-11T14:41:29.748491+0000","last_fullsized":"2024-09-11T14:41:29.748491+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"19'33","reported_seq":45,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.108110+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:41:29.108110+0000","last_peered":"2024-09-11T14:41:29.108110+0000","last_clean":"2024-09-11T14:41:29.108110+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:41:29.108110+0000","last_undegraded":"2024-09-11T14:41:29.108110+0000","last_fullsized":"2024-09-11T14:41:29.108110+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2162688,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":2112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"19'41","reported_seq":53,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.141380+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:41:29.141380+0000","last_peered":"2024-09-11T14:41:29.141380+0000","last_clean":"2024-09-11T14:41:29.141380+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:41:29.141380+0000","last_undegraded":"2024-09-11T14:41:29.141380+0000","last_fullsized":"2024-09-11T14:41:29.141380+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2686976,"num_objects":41,"num_object_clones":0,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41,"num_write_kb":2624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":97,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:20.852495+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:41:20.852495+0000","last_peered":"2024-09-11T14:41:20.852495+0000","last_clean":"2024-09-11T14:41:20.852495+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:41:20.852495+0000","last_undegraded":"2024-09-11T14:41:20.852495+0000","last_fullsized":"2024-09-11T14:41:20.852495+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:20.858684+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:41:20.858684+0000","last_peered":"2024-09-11T14:41:20.858684+0000","last_clean":"2024-09-11T14:41:20.858684+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:41:20.858684+0000","last_undegraded":"2024-09-11T14:41:20.858684+0000","last_fullsized":"2024-09-11T14:41:20.858684+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"19'42","reported_seq":54,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.590263+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:41:29.590263+0000","last_peered":"2024-09-11T14:41:29.590263+0000","last_clean":"2024-09-11T14:41:29.590263+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:41:29.590263+0000","last_undegraded":"2024-09-11T14:41:29.590263+0000","last_fullsized":"2024-09-11T14:41:29.590263+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2752512,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42,"num_write_kb":2688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":25,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:20.854776+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:41:20.854776+0000","last_peered":"2024-09-11T14:41:20.854776+0000","last_clean":"2024-09-11T14:41:20.854776+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:41:20.854776+0000","last_undegraded":"2024-09-11T14:41:20.854776+0000","last_fullsized":"2024-09-11T14:41:20.854776+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"19'41","reported_seq":53,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.353129+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:41:29.353129+0000","last_peered":"2024-09-11T14:41:29.353129+0000","last_clean":"2024-09-11T14:41:29.353129+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:41:29.353129+0000","last_undegraded":"2024-09-11T14:41:29.353129+0000","last_fullsized":"2024-09-11T14:41:29.353129+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2686976,"num_objects":41,"num_object_clones":0,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41,"num_write_kb":2624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:20.851661+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:41:20.851661+0000","last_peered":"2024-09-11T14:41:20.851661+0000","last_clean":"2024-09-11T14:41:20.851661+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:41:20.851661+0000","last_undegraded":"2024-09-11T14:41:20.851661+0000","last_fullsized":"2024-09-11T14:41:20.851661+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"19'56","reported_seq":68,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.523300+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:41:29.523300+0000","last_peered":"2024-09-11T14:41:29.523300+0000","last_clean":"2024-09-11T14:41:29.523300+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:41:29.523300+0000","last_undegraded":"2024-09-11T14:41:29.523300+0000","last_fullsized":"2024-09-11T14:41:29.523300+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3670016,"num_objects":56,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56,"num_write_kb":3584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:20.855755+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:41:20.855755+0000","last_peered":"2024-09-11T14:41:20.855755+0000","last_clean":"2024-09-11T14:41:20.855755+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:41:20.855755+0000","last_undegraded":"2024-09-11T14:41:20.855755+0000","last_fullsized":"2024-09-11T14:41:20.855755+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"19'47","reported_seq":59,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:30.291721+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:41:30.291721+0000","last_peered":"2024-09-11T14:41:30.291721+0000","last_clean":"2024-09-11T14:41:30.291721+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:41:30.291721+0000","last_undegraded":"2024-09-11T14:41:30.291721+0000","last_fullsized":"2024-09-11T14:41:30.291721+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3080192,"num_objects":47,"num_object_clones":0,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":3008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"19'45","reported_seq":57,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.687569+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:41:29.687569+0000","last_peered":"2024-09-11T14:41:29.687569+0000","last_clean":"2024-09-11T14:41:29.687569+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:41:29.687569+0000","last_undegraded":"2024-09-11T14:41:29.687569+0000","last_fullsized":"2024-09-11T14:41:29.687569+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2949120,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":2880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:20.855868+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:41:20.855868+0000","last_peered":"2024-09-11T14:41:20.855868+0000","last_clean":"2024-09-11T14:41:20.855868+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:41:20.855868+0000","last_undegraded":"2024-09-11T14:41:20.855868+0000","last_fullsized":"2024-09-11T14:41:20.855868+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"19'42","reported_seq":54,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:30.533620+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:41:30.533620+0000","last_peered":"2024-09-11T14:41:30.533620+0000","last_clean":"2024-09-11T14:41:30.533620+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:41:30.533620+0000","last_undegraded":"2024-09-11T14:41:30.533620+0000","last_fullsized":"2024-09-11T14:41:30.533620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2752512,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42,"num_write_kb":2688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:20.850469+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:41:20.850469+0000","last_peered":"2024-09-11T14:41:20.850469+0000","last_clean":"2024-09-11T14:41:20.850469+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:41:20.850469+0000","last_undegraded":"2024-09-11T14:41:20.850469+0000","last_fullsized":"2024-09-11T14:41:20.850469+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"19'51","reported_seq":63,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:29.573151+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:41:29.573151+0000","last_peered":"2024-09-11T14:41:29.573151+0000","last_clean":"2024-09-11T14:41:29.573151+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:41:29.573151+0000","last_undegraded":"2024-09-11T14:41:29.573151+0000","last_fullsized":"2024-09-11T14:41:29.573151+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3342336,"num_objects":51,"num_object_clones":0,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51,"num_write_kb":3264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":25,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:20.856195+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:41:20.856195+0000","last_peered":"2024-09-11T14:41:20.856195+0000","last_clean":"2024-09-11T14:41:20.856195+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:41:20.856195+0000","last_undegraded":"2024-09-11T14:41:20.856195+0000","last_fullsized":"2024-09-11T14:41:20.856195+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"19'60","reported_seq":72,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:30.692828+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:41:30.692828+0000","last_peered":"2024-09-11T14:41:30.692828+0000","last_clean":"2024-09-11T14:41:30.692828+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:41:30.692828+0000","last_undegraded":"2024-09-11T14:41:30.692828+0000","last_fullsized":"2024-09-11T14:41:30.692828+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3932160,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60,"num_write_kb":3840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T14:41:20.858443+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:41:20.858443+0000","last_peered":"2024-09-11T14:41:20.858443+0000","last_clean":"2024-09-11T14:41:20.858443+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:41:20.858443+0000","last_undegraded":"2024-09-11T14:41:20.858443+0000","last_fullsized":"2024-09-11T14:41:20.858443+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":48824320,"num_objects":745,"num_object_clones":0,"num_object_copies":2235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":745,"num_write_kb":47680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":213911605248,"internally_reserved":0,"allocated":299888640,"data_stored":299888640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":745,"ondisk_log_size":745,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":213911736320,"internally_reserved":0,"allocated":299757568,"data_stored":299757568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53467095040,"internally_reserved":0,"allocated":85778432,"data_stored":85778432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":68908,"kb_used_data":68908,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52228820,"statfs":{"total":53552873472,"available":53482311680,"internally_reserved":0,"allocated":70561792,"data_stored":70561792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705688,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":72848,"kb_used_data":72848,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52224880,"statfs":{"total":53552873472,"available":53478277120,"internally_reserved":0,"allocated":74596352,"data_stored":74596352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705688,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":59908,"kb_used_data":59908,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52237820,"statfs":{"total":53552873472,"available":53491527680,"internally_reserved":0,"allocated":61345792,"data_stored":61345792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705688,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":54140,"kb_used_data":54140,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52243588,"statfs":{"total":53552873472,"available":53497434112,"internally_reserved":0,"allocated":55439360,"data_stored":55439360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53467095040,"internally_reserved":0,"allocated":85778432,"data_stored":85778432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53490012160,"internally_reserved":0,"allocated":62861312,"data_stored":62861312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53480869888,"internally_reserved":0,"allocated":72003584,"data_stored":72003584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53473759232,"internally_reserved":0,"allocated":79114240,"data_stored":79114240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53467095040,"internally_reserved":0,"allocated":85778432,"data_stored":85778432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":53490012160,"internally_reserved":0,"allocated":62861312,"data_stored":62861312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":53480869888,"internally_reserved":0,"allocated":72003584,"data_stored":72003584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":53473628160,"internally_reserved":0,"allocated":79245312,"data_stored":79245312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":53467095040,"internally_reserved":0,"allocated":85778432,"data_stored":85778432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:41:35.904 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 15 16 1281 1265 5.27036 6 0.0744186 0.188539 2024-09-11T14:41:36.904 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 16 16 1379 1363 5.32374 6.125 0.219762 0.186617 2024-09-11T14:41:37.249 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:41:37.406 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:37.402+0000 7f92e3951640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:37.414 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:37.412+0000 7f92e3951640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:37.904 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 17 16 1483 1467 5.39289 6.5 0.153247 0.184273 2024-09-11T14:41:38.905 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 18 16 1573 1557 5.40575 5.625 0.17134 0.183878 2024-09-11T14:41:39.877 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:41:39.905 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 19 16 1676 1660 5.45998 6.4375 0.0962135 0.182545 2024-09-11T14:41:40.057 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:40.052+0000 7f5bc6888640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:40.064 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:40.063+0000 7f5bc6888640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:40.568 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:41:40.568 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:41:40.568 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:41:40.741 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:40.732+0000 7f0676151640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:40.747 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:40.746+0000 7f0676151640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:40.905 INFO:tasks.radosbench.radosbench.0.smithi181.stdout:2024-09-11T14:41:40.905127+0000 min lat: 0.0249053 max lat: 1.01958 avg lat: 0.180803 2024-09-11T14:41:40.905 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T14:41:40.906 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 20 16 1775 1759 5.4963 6.1875 0.0685926 0.180803 2024-09-11T14:41:41.123 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:41:41.124 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:41:41.148 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-09-11T14:41:40.711929+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103350835,"num_objects":1572,"num_object_clones":0,"num_object_copies":4716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":1649,"num_write_kb":101460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1623,"ondisk_log_size":1623,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":575852,"kb_used_data":575852,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":208615060,"statfs":{"total":214211493888,"available":213621821440,"internally_reserved":0,"allocated":589672448,"data_stored":589672448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":53936128,"num_objects":823,"num_object_clones":0,"num_object_copies":2469,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":823,"num_write_kb":52672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001544"},"pg_stats":[{"pgid":"3.f","version":"23'89","reported_seq":113,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:39.390880+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:41:39.390880+0000","last_peered":"2024-09-11T14:41:39.390880+0000","last_clean":"2024-09-11T14:41:39.390880+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:41:39.390880+0000","last_undegraded":"2024-09-11T14:41:39.390880+0000","last_fullsized":"2024-09-11T14:41:39.390880+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5832704,"num_objects":89,"num_object_clones":0,"num_object_copies":267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89,"num_write_kb":5696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"23'91","reported_seq":115,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.986709+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:41:34.986709+0000","last_peered":"2024-09-11T14:41:34.986709+0000","last_clean":"2024-09-11T14:41:34.986709+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:41:34.986709+0000","last_undegraded":"2024-09-11T14:41:34.986709+0000","last_fullsized":"2024-09-11T14:41:34.986709+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5963776,"num_objects":91,"num_object_clones":0,"num_object_copies":273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91,"num_write_kb":5824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"23'104","reported_seq":128,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:39.683858+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:41:39.683858+0000","last_peered":"2024-09-11T14:41:39.683858+0000","last_clean":"2024-09-11T14:41:39.683858+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:41:39.683858+0000","last_undegraded":"2024-09-11T14:41:39.683858+0000","last_fullsized":"2024-09-11T14:41:39.683858+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6815744,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":104,"num_write_kb":6656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"23'110","reported_seq":134,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:39.109322+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:41:39.109322+0000","last_peered":"2024-09-11T14:41:39.109322+0000","last_clean":"2024-09-11T14:41:39.109322+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:41:39.109322+0000","last_undegraded":"2024-09-11T14:41:39.109322+0000","last_fullsized":"2024-09-11T14:41:39.109322+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7208960,"num_objects":110,"num_object_clones":0,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110,"num_write_kb":7040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"23'108","reported_seq":132,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:39.701938+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:41:39.701938+0000","last_peered":"2024-09-11T14:41:39.701938+0000","last_clean":"2024-09-11T14:41:39.701938+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:41:39.701938+0000","last_undegraded":"2024-09-11T14:41:39.701938+0000","last_fullsized":"2024-09-11T14:41:39.701938+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7077888,"num_objects":108,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":108,"num_write_kb":6912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"23'113","reported_seq":137,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:39.673013+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:41:39.673013+0000","last_peered":"2024-09-11T14:41:39.673013+0000","last_clean":"2024-09-11T14:41:39.673013+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:41:39.673013+0000","last_undegraded":"2024-09-11T14:41:39.673013+0000","last_fullsized":"2024-09-11T14:41:39.673013+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7405568,"num_objects":113,"num_object_clones":0,"num_object_copies":339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":113,"num_write_kb":7232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"23'95","reported_seq":119,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:39.474052+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:41:39.474052+0000","last_peered":"2024-09-11T14:41:39.474052+0000","last_clean":"2024-09-11T14:41:39.474052+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:41:39.474052+0000","last_undegraded":"2024-09-11T14:41:39.474052+0000","last_fullsized":"2024-09-11T14:41:39.474052+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6225920,"num_objects":95,"num_object_clones":0,"num_object_copies":285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95,"num_write_kb":6080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"23'92","reported_seq":116,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:39.499492+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:41:39.499492+0000","last_peered":"2024-09-11T14:41:39.499492+0000","last_clean":"2024-09-11T14:41:39.499492+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:41:39.499492+0000","last_undegraded":"2024-09-11T14:41:39.499492+0000","last_fullsized":"2024-09-11T14:41:39.499492+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6029312,"num_objects":92,"num_object_clones":0,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92,"num_write_kb":5888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":109,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:33.882331+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:41:33.882331+0000","last_peered":"2024-09-11T14:41:33.882331+0000","last_clean":"2024-09-11T14:41:33.882331+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:41:33.882331+0000","last_undegraded":"2024-09-11T14:41:33.882331+0000","last_fullsized":"2024-09-11T14:41:33.882331+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":38,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.165313+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:41:34.165313+0000","last_peered":"2024-09-11T14:41:34.165313+0000","last_clean":"2024-09-11T14:41:34.165313+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:41:34.165313+0000","last_undegraded":"2024-09-11T14:41:34.165313+0000","last_fullsized":"2024-09-11T14:41:34.165313+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"23'102","reported_seq":126,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:39.633645+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:41:39.633645+0000","last_peered":"2024-09-11T14:41:39.633645+0000","last_clean":"2024-09-11T14:41:39.633645+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:41:39.633645+0000","last_undegraded":"2024-09-11T14:41:39.633645+0000","last_fullsized":"2024-09-11T14:41:39.633645+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6684672,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":6528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.022265+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:41:34.022265+0000","last_peered":"2024-09-11T14:41:34.022265+0000","last_clean":"2024-09-11T14:41:34.022265+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:41:34.022265+0000","last_undegraded":"2024-09-11T14:41:34.022265+0000","last_fullsized":"2024-09-11T14:41:34.022265+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"23'100","reported_seq":124,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:39.801306+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:41:39.801306+0000","last_peered":"2024-09-11T14:41:39.801306+0000","last_clean":"2024-09-11T14:41:39.801306+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:41:39.801306+0000","last_undegraded":"2024-09-11T14:41:39.801306+0000","last_fullsized":"2024-09-11T14:41:39.801306+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6553600,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":6400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.163617+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:41:34.163617+0000","last_peered":"2024-09-11T14:41:34.163617+0000","last_clean":"2024-09-11T14:41:34.163617+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:41:34.163617+0000","last_undegraded":"2024-09-11T14:41:34.163617+0000","last_fullsized":"2024-09-11T14:41:34.163617+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"23'129","reported_seq":153,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:39.663864+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:41:39.663864+0000","last_peered":"2024-09-11T14:41:39.663864+0000","last_clean":"2024-09-11T14:41:39.663864+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:41:39.663864+0000","last_undegraded":"2024-09-11T14:41:39.663864+0000","last_fullsized":"2024-09-11T14:41:39.663864+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8454144,"num_objects":129,"num_object_clones":0,"num_object_copies":387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":8256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":39,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.025108+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:41:34.025108+0000","last_peered":"2024-09-11T14:41:34.025108+0000","last_clean":"2024-09-11T14:41:34.025108+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:41:34.025108+0000","last_undegraded":"2024-09-11T14:41:34.025108+0000","last_fullsized":"2024-09-11T14:41:34.025108+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"23'67","reported_seq":91,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:35.570492+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:41:35.570492+0000","last_peered":"2024-09-11T14:41:35.570492+0000","last_clean":"2024-09-11T14:41:35.570492+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:41:35.570492+0000","last_undegraded":"2024-09-11T14:41:35.570492+0000","last_fullsized":"2024-09-11T14:41:35.570492+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4390912,"num_objects":67,"num_object_clones":0,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67,"num_write_kb":4288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"23'100","reported_seq":124,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:39.757414+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:41:39.757414+0000","last_peered":"2024-09-11T14:41:39.757414+0000","last_clean":"2024-09-11T14:41:39.757414+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:41:39.757414+0000","last_undegraded":"2024-09-11T14:41:39.757414+0000","last_fullsized":"2024-09-11T14:41:39.757414+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6553600,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":6400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.028115+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:41:34.028115+0000","last_peered":"2024-09-11T14:41:34.028115+0000","last_clean":"2024-09-11T14:41:34.028115+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:41:34.028115+0000","last_undegraded":"2024-09-11T14:41:34.028115+0000","last_fullsized":"2024-09-11T14:41:34.028115+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"23'67","reported_seq":91,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:35.694656+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:41:35.694656+0000","last_peered":"2024-09-11T14:41:35.694656+0000","last_clean":"2024-09-11T14:41:35.694656+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:41:35.694656+0000","last_undegraded":"2024-09-11T14:41:35.694656+0000","last_fullsized":"2024-09-11T14:41:35.694656+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4390912,"num_objects":67,"num_object_clones":0,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67,"num_write_kb":4288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:33.885106+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:41:33.885106+0000","last_peered":"2024-09-11T14:41:33.885106+0000","last_clean":"2024-09-11T14:41:33.885106+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:41:33.885106+0000","last_undegraded":"2024-09-11T14:41:33.885106+0000","last_fullsized":"2024-09-11T14:41:33.885106+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"23'111","reported_seq":135,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:39.353468+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:41:39.353468+0000","last_peered":"2024-09-11T14:41:39.353468+0000","last_clean":"2024-09-11T14:41:39.353468+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:41:39.353468+0000","last_undegraded":"2024-09-11T14:41:39.353468+0000","last_fullsized":"2024-09-11T14:41:39.353468+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7274496,"num_objects":111,"num_object_clones":0,"num_object_copies":333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":111,"num_write_kb":7104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.025417+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:41:34.025417+0000","last_peered":"2024-09-11T14:41:34.025417+0000","last_clean":"2024-09-11T14:41:34.025417+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:41:34.025417+0000","last_undegraded":"2024-09-11T14:41:34.025417+0000","last_fullsized":"2024-09-11T14:41:34.025417+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"23'90","reported_seq":114,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:35.715673+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:41:35.715673+0000","last_peered":"2024-09-11T14:41:35.715673+0000","last_clean":"2024-09-11T14:41:35.715673+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:41:35.715673+0000","last_undegraded":"2024-09-11T14:41:35.715673+0000","last_fullsized":"2024-09-11T14:41:35.715673+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5898240,"num_objects":90,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":5760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.169641+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:41:34.169641+0000","last_peered":"2024-09-11T14:41:34.169641+0000","last_clean":"2024-09-11T14:41:34.169641+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:41:34.169641+0000","last_undegraded":"2024-09-11T14:41:34.169641+0000","last_fullsized":"2024-09-11T14:41:34.169641+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":102760448,"num_objects":1568,"num_object_clones":0,"num_object_copies":4704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1568,"num_write_kb":100352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":213584003072,"internally_reserved":0,"allocated":627490816,"data_stored":627490816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1568,"ondisk_log_size":1568,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":213584003072,"internally_reserved":0,"allocated":627490816,"data_stored":627490816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53359874048,"internally_reserved":0,"allocated":192999424,"data_stored":192999424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":108052,"kb_used_data":108052,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52189676,"statfs":{"total":53552873472,"available":53442228224,"internally_reserved":0,"allocated":110645248,"data_stored":110645248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705698,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":177248,"kb_used_data":177248,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52120480,"statfs":{"total":53552873472,"available":53371371520,"internally_reserved":0,"allocated":181501952,"data_stored":181501952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705698,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":153620,"kb_used_data":153620,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52144108,"statfs":{"total":53552873472,"available":53395566592,"internally_reserved":0,"allocated":157306880,"data_stored":157306880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705698,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":136932,"kb_used_data":136932,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52160796,"statfs":{"total":53552873472,"available":53412655104,"internally_reserved":0,"allocated":140218368,"data_stored":140218368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53359874048,"internally_reserved":0,"allocated":192999424,"data_stored":192999424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53403705344,"internally_reserved":0,"allocated":149168128,"data_stored":149168128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53386641408,"internally_reserved":0,"allocated":166232064,"data_stored":166232064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53433782272,"internally_reserved":0,"allocated":119091200,"data_stored":119091200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53359874048,"internally_reserved":0,"allocated":192999424,"data_stored":192999424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":53403705344,"internally_reserved":0,"allocated":149168128,"data_stored":149168128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":53386641408,"internally_reserved":0,"allocated":166232064,"data_stored":166232064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":53433782272,"internally_reserved":0,"allocated":119091200,"data_stored":119091200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":53359874048,"internally_reserved":0,"allocated":192999424,"data_stored":192999424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:41:41.905 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 21 15 1885 1870 5.56489 6.9375 0.0861329 0.178982 2024-09-11T14:41:42.346 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:41:42.517 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:42.513+0000 7f06fb791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:42.524 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:42.521+0000 7f06fb791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:42.905 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 22 16 1981 1965 5.5818 5.9375 0.483905 0.178324 2024-09-11T14:41:43.905 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 23 16 2079 2063 5.6054 6.125 0.169125 0.177658 2024-09-11T14:41:44.872 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:41:44.905 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 24 16 2182 2166 5.64005 6.4375 0.0932744 0.176553 2024-09-11T14:41:45.031 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:45.027+0000 7f0e7d391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:45.037 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:45.036+0000 7f0e7d391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:45.905 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 25 16 2281 2265 5.66192 6.1875 0.0695176 0.175959 2024-09-11T14:41:46.152 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:41:46.152 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:41:46.152 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-11T14:41:46.152 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-11T14:41:46.304 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:46.295+0000 7eff46791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:46.309 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:46.308+0000 7eff46791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:46.342 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:41:46.342 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:41:46.343 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:41:46.343 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:41:46.343 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:41:46.343 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:41:46.343 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:41:46.343 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:41:46.343 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:41:46.343 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:41:46.343 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:41:46.343 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:41:46.346 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:41:46.347 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:41:46.347 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-11T14:41:46.499 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:46.486+0000 7efea647f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:46.503 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:46.502+0000 7efea647f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:46.587 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:41:46.587 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:41:46.587 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:41:46.588 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:41:46.588 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:41:46.588 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:41:46.588 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:41:46.588 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:41:46.588 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:41:46.588 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:41:46.588 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:41:46.588 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:41:46.591 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:41:46.592 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:41:46.592 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-11T14:41:46.741 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:46.733+0000 7f4bb6551640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:46.748 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:46.747+0000 7f4bb6551640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:46.883 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:41:46.883 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:41:46.883 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:41:46.884 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:41:46.884 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:41:46.884 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:41:46.884 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:41:46.884 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:41:46.884 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:41:46.884 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:41:46.884 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:41:46.884 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:41:46.888 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:41:46.888 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:41:46.888 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-11T14:41:46.906 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 26 16 2386 2370 5.69653 6.5625 0.106272 0.175031 2024-09-11T14:41:47.044 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:47.036+0000 7fa3d4a84640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:47.050 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:47.049+0000 7fa3d4a84640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:47.179 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:41:47.179 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:41:47.179 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:41:47.179 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:41:47.179 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:41:47.180 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:41:47.180 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:41:47.180 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:41:47.180 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:41:47.180 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:41:47.180 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:41:47.180 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:41:47.183 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:41:47.183 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:41:47.323 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:41:47.465 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:47.462+0000 7fb8e4791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:47.470 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:47.468+0000 7fb8e4791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:47.906 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 27 16 2480 2464 5.70312 5.875 0.0657075 0.174199 2024-09-11T14:41:48.906 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 28 16 2582 2566 5.7271 6.375 0.159701 0.174037 2024-09-11T14:41:49.802 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:41:49.906 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 29 16 2683 2667 5.74726 6.3125 0.0533526 0.17318 2024-09-11T14:41:49.960 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:49.956+0000 7fec24cac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:49.966 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:49.965+0000 7fec24cac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:50.906 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 30 16 2788 2772 5.77442 6.5625 0.0899816 0.172512 2024-09-11T14:41:51.906 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 31 16 2895 2879 5.80385 6.6875 0.225504 0.171745 2024-09-11T14:41:52.185 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:41:52.185 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:41:52.185 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:41:52.341 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:52.334+0000 7f6e7f191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:52.346 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:52.345+0000 7f6e7f191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:52.440 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:41:52.591 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:52.586+0000 7fa8fb0ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:52.597 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:52.596+0000 7fa8fb0ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:52.631 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:41:52.632 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:41:52.651 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-09-11T14:41:50.713898+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":169017907,"num_objects":2574,"num_object_clones":0,"num_object_copies":7722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":2651,"num_write_kb":165588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2625,"ondisk_log_size":2625,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":964424,"kb_used_data":964424,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":208226488,"statfs":{"total":214211493888,"available":213223923712,"internally_reserved":0,"allocated":987570176,"data_stored":987570176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":89784320,"num_objects":1370,"num_object_clones":0,"num_object_copies":4110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1370,"num_write_kb":87680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002364"},"pg_stats":[{"pgid":"3.f","version":"23'148","reported_seq":172,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:48.809028+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:41:48.809028+0000","last_peered":"2024-09-11T14:41:48.809028+0000","last_clean":"2024-09-11T14:41:48.809028+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:41:48.809028+0000","last_undegraded":"2024-09-11T14:41:48.809028+0000","last_fullsized":"2024-09-11T14:41:48.809028+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9699328,"num_objects":148,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":148,"num_write_kb":9472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"23'163","reported_seq":187,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:45.802651+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:41:45.802651+0000","last_peered":"2024-09-11T14:41:45.802651+0000","last_clean":"2024-09-11T14:41:45.802651+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:41:45.802651+0000","last_undegraded":"2024-09-11T14:41:45.802651+0000","last_fullsized":"2024-09-11T14:41:45.802651+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10682368,"num_objects":163,"num_object_clones":0,"num_object_copies":489,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":163,"num_write_kb":10432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"23'165","reported_seq":189,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:49.771740+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:41:49.771740+0000","last_peered":"2024-09-11T14:41:49.771740+0000","last_clean":"2024-09-11T14:41:49.771740+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:41:49.771740+0000","last_undegraded":"2024-09-11T14:41:49.771740+0000","last_fullsized":"2024-09-11T14:41:49.771740+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10813440,"num_objects":165,"num_object_clones":0,"num_object_copies":495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":10560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"23'177","reported_seq":201,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:49.610242+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:41:49.610242+0000","last_peered":"2024-09-11T14:41:49.610242+0000","last_clean":"2024-09-11T14:41:49.610242+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:41:49.610242+0000","last_undegraded":"2024-09-11T14:41:49.610242+0000","last_fullsized":"2024-09-11T14:41:49.610242+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11599872,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":177,"num_write_kb":11328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"23'173","reported_seq":197,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:49.416141+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:41:49.416141+0000","last_peered":"2024-09-11T14:41:49.416141+0000","last_clean":"2024-09-11T14:41:49.416141+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:41:49.416141+0000","last_undegraded":"2024-09-11T14:41:49.416141+0000","last_fullsized":"2024-09-11T14:41:49.416141+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11337728,"num_objects":173,"num_object_clones":0,"num_object_copies":519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":173,"num_write_kb":11072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"23'173","reported_seq":197,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:49.670767+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:41:49.670767+0000","last_peered":"2024-09-11T14:41:49.670767+0000","last_clean":"2024-09-11T14:41:49.670767+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:41:49.670767+0000","last_undegraded":"2024-09-11T14:41:49.670767+0000","last_fullsized":"2024-09-11T14:41:49.670767+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11337728,"num_objects":173,"num_object_clones":0,"num_object_copies":519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":173,"num_write_kb":11072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"23'167","reported_seq":191,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:49.677874+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:41:49.677874+0000","last_peered":"2024-09-11T14:41:49.677874+0000","last_clean":"2024-09-11T14:41:49.677874+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:41:49.677874+0000","last_undegraded":"2024-09-11T14:41:49.677874+0000","last_fullsized":"2024-09-11T14:41:49.677874+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10944512,"num_objects":167,"num_object_clones":0,"num_object_copies":501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":167,"num_write_kb":10688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"23'165","reported_seq":189,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:49.399759+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:41:49.399759+0000","last_peered":"2024-09-11T14:41:49.399759+0000","last_clean":"2024-09-11T14:41:49.399759+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:41:49.399759+0000","last_undegraded":"2024-09-11T14:41:49.399759+0000","last_fullsized":"2024-09-11T14:41:49.399759+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10813440,"num_objects":165,"num_object_clones":0,"num_object_copies":495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":10560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":109,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:33.882331+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:41:33.882331+0000","last_peered":"2024-09-11T14:41:33.882331+0000","last_clean":"2024-09-11T14:41:33.882331+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:41:33.882331+0000","last_undegraded":"2024-09-11T14:41:33.882331+0000","last_fullsized":"2024-09-11T14:41:33.882331+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":38,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.165313+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:41:34.165313+0000","last_peered":"2024-09-11T14:41:34.165313+0000","last_clean":"2024-09-11T14:41:34.165313+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:41:34.165313+0000","last_undegraded":"2024-09-11T14:41:34.165313+0000","last_fullsized":"2024-09-11T14:41:34.165313+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"23'163","reported_seq":187,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:49.495134+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:41:49.495134+0000","last_peered":"2024-09-11T14:41:49.495134+0000","last_clean":"2024-09-11T14:41:49.495134+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:41:49.495134+0000","last_undegraded":"2024-09-11T14:41:49.495134+0000","last_fullsized":"2024-09-11T14:41:49.495134+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10682368,"num_objects":163,"num_object_clones":0,"num_object_copies":489,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":163,"num_write_kb":10432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.022265+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:41:34.022265+0000","last_peered":"2024-09-11T14:41:34.022265+0000","last_clean":"2024-09-11T14:41:34.022265+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:41:34.022265+0000","last_undegraded":"2024-09-11T14:41:34.022265+0000","last_fullsized":"2024-09-11T14:41:34.022265+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"23'145","reported_seq":169,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:49.357803+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:41:49.357803+0000","last_peered":"2024-09-11T14:41:49.357803+0000","last_clean":"2024-09-11T14:41:49.357803+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:41:49.357803+0000","last_undegraded":"2024-09-11T14:41:49.357803+0000","last_fullsized":"2024-09-11T14:41:49.357803+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9502720,"num_objects":145,"num_object_clones":0,"num_object_copies":435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":145,"num_write_kb":9280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.163617+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:41:34.163617+0000","last_peered":"2024-09-11T14:41:34.163617+0000","last_clean":"2024-09-11T14:41:34.163617+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:41:34.163617+0000","last_undegraded":"2024-09-11T14:41:34.163617+0000","last_fullsized":"2024-09-11T14:41:34.163617+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"23'187","reported_seq":211,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:49.608373+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:41:49.608373+0000","last_peered":"2024-09-11T14:41:49.608373+0000","last_clean":"2024-09-11T14:41:49.608373+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:41:49.608373+0000","last_undegraded":"2024-09-11T14:41:49.608373+0000","last_fullsized":"2024-09-11T14:41:49.608373+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12255232,"num_objects":187,"num_object_clones":0,"num_object_copies":561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":187,"num_write_kb":11968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":39,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.025108+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:41:34.025108+0000","last_peered":"2024-09-11T14:41:34.025108+0000","last_clean":"2024-09-11T14:41:34.025108+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:41:34.025108+0000","last_undegraded":"2024-09-11T14:41:34.025108+0000","last_fullsized":"2024-09-11T14:41:34.025108+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"23'135","reported_seq":159,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:45.684590+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:41:45.684590+0000","last_peered":"2024-09-11T14:41:45.684590+0000","last_clean":"2024-09-11T14:41:45.684590+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:41:45.684590+0000","last_undegraded":"2024-09-11T14:41:45.684590+0000","last_fullsized":"2024-09-11T14:41:45.684590+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8847360,"num_objects":135,"num_object_clones":0,"num_object_copies":405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":8640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"23'169","reported_seq":193,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:49.536952+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:41:49.536952+0000","last_peered":"2024-09-11T14:41:49.536952+0000","last_clean":"2024-09-11T14:41:49.536952+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:41:49.536952+0000","last_undegraded":"2024-09-11T14:41:49.536952+0000","last_fullsized":"2024-09-11T14:41:49.536952+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11075584,"num_objects":169,"num_object_clones":0,"num_object_copies":507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":169,"num_write_kb":10816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.028115+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:41:34.028115+0000","last_peered":"2024-09-11T14:41:34.028115+0000","last_clean":"2024-09-11T14:41:34.028115+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:41:34.028115+0000","last_undegraded":"2024-09-11T14:41:34.028115+0000","last_fullsized":"2024-09-11T14:41:34.028115+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"23'123","reported_seq":147,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:44.682163+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:41:44.682163+0000","last_peered":"2024-09-11T14:41:44.682163+0000","last_clean":"2024-09-11T14:41:44.682163+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:41:44.682163+0000","last_undegraded":"2024-09-11T14:41:44.682163+0000","last_fullsized":"2024-09-11T14:41:44.682163+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8060928,"num_objects":123,"num_object_clones":0,"num_object_copies":369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":7872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:33.885106+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:41:33.885106+0000","last_peered":"2024-09-11T14:41:33.885106+0000","last_clean":"2024-09-11T14:41:33.885106+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:41:33.885106+0000","last_undegraded":"2024-09-11T14:41:33.885106+0000","last_fullsized":"2024-09-11T14:41:33.885106+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"23'176","reported_seq":200,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:49.435430+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:41:49.435430+0000","last_peered":"2024-09-11T14:41:49.435430+0000","last_clean":"2024-09-11T14:41:49.435430+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:41:49.435430+0000","last_undegraded":"2024-09-11T14:41:49.435430+0000","last_fullsized":"2024-09-11T14:41:49.435430+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11534336,"num_objects":176,"num_object_clones":0,"num_object_copies":528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":176,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.025417+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:41:34.025417+0000","last_peered":"2024-09-11T14:41:34.025417+0000","last_clean":"2024-09-11T14:41:34.025417+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:41:34.025417+0000","last_undegraded":"2024-09-11T14:41:34.025417+0000","last_fullsized":"2024-09-11T14:41:34.025417+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"23'141","reported_seq":165,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:45.600920+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:41:45.600920+0000","last_peered":"2024-09-11T14:41:45.600920+0000","last_clean":"2024-09-11T14:41:45.600920+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:41:45.600920+0000","last_undegraded":"2024-09-11T14:41:45.600920+0000","last_fullsized":"2024-09-11T14:41:45.600920+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9240576,"num_objects":141,"num_object_clones":0,"num_object_copies":423,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":9024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.169641+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:41:34.169641+0000","last_peered":"2024-09-11T14:41:34.169641+0000","last_clean":"2024-09-11T14:41:34.169641+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:41:34.169641+0000","last_undegraded":"2024-09-11T14:41:34.169641+0000","last_fullsized":"2024-09-11T14:41:34.169641+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":168427520,"num_objects":2570,"num_object_clones":0,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2570,"num_write_kb":164480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":213186891776,"internally_reserved":0,"allocated":1024602112,"data_stored":1024602112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2570,"ondisk_log_size":2570,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":213187022848,"internally_reserved":0,"allocated":1024471040,"data_stored":1024471040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53244182528,"internally_reserved":0,"allocated":308690944,"data_stored":308690944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672999,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":198152,"kb_used_data":198152,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52099576,"statfs":{"total":53552873472,"available":53349965824,"internally_reserved":0,"allocated":202907648,"data_stored":202907648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705708,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":290148,"kb_used_data":290148,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52007580,"statfs":{"total":53552873472,"available":53255761920,"internally_reserved":0,"allocated":297111552,"data_stored":297111552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705708,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":250856,"kb_used_data":250856,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52046872,"statfs":{"total":53552873472,"available":53295996928,"internally_reserved":0,"allocated":256876544,"data_stored":256876544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705708,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":225268,"kb_used_data":225268,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52072460,"statfs":{"total":53552873472,"available":53322199040,"internally_reserved":0,"allocated":230674432,"data_stored":230674432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53244182528,"internally_reserved":0,"allocated":308690944,"data_stored":308690944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53313495040,"internally_reserved":0,"allocated":239378432,"data_stored":239378432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53286350848,"internally_reserved":0,"allocated":266522624,"data_stored":266522624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53342994432,"internally_reserved":0,"allocated":209879040,"data_stored":209879040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53244182528,"internally_reserved":0,"allocated":308690944,"data_stored":308690944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":53313495040,"internally_reserved":0,"allocated":239378432,"data_stored":239378432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":53286350848,"internally_reserved":0,"allocated":266522624,"data_stored":266522624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":53342994432,"internally_reserved":0,"allocated":209879040,"data_stored":209879040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":53244051456,"internally_reserved":0,"allocated":308822016,"data_stored":308822016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:41:52.906 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 32 16 2996 2980 5.81973 6.3125 0.0880199 0.171396 2024-09-11T14:41:53.906 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 33 16 3093 3077 5.82707 6.0625 0.148395 0.171031 2024-09-11T14:41:54.906 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 34 16 3206 3190 5.86338 7.0625 0.0644592 0.170152 2024-09-11T14:41:55.010 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:41:55.153 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:55.149+0000 7fe237d11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:55.158 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:55.157+0000 7fe237d11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:55.906 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 35 16 3314 3298 5.8887 6.75 0.210214 0.169444 2024-09-11T14:41:56.907 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 36 16 3427 3411 5.92128 7.0625 0.114598 0.168555 2024-09-11T14:41:57.630 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:41:57.654 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:41:57.654 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:41:57.654 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:41:57.783 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:57.777+0000 7f4e08b08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:57.789 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:57.788+0000 7f4e08b08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:57.873 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:57.866+0000 7f20b30bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:57.882 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:41:57.878+0000 7f20b30bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:41:57.907 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 37 16 3528 3512 5.93184 6.3125 0.232668 0.168168 2024-09-11T14:41:58.137 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:41:58.138 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:41:58.155 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-09-11T14:41:56.715195+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":211550771,"num_objects":3223,"num_object_clones":0,"num_object_copies":9669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":3300,"num_write_kb":207124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3274,"ondisk_log_size":3274,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":1207248,"kb_used_data":1207248,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":207983664,"statfs":{"total":214211493888,"available":212975271936,"internally_reserved":0,"allocated":1236221952,"data_stored":1236221952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":100204544,"num_objects":1529,"num_object_clones":0,"num_object_copies":4587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1529,"num_write_kb":97856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002542"},"pg_stats":[{"pgid":"3.f","version":"23'191","reported_seq":215,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:54.632734+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:41:54.632734+0000","last_peered":"2024-09-11T14:41:54.632734+0000","last_clean":"2024-09-11T14:41:54.632734+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:41:54.632734+0000","last_undegraded":"2024-09-11T14:41:54.632734+0000","last_fullsized":"2024-09-11T14:41:54.632734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12517376,"num_objects":191,"num_object_clones":0,"num_object_copies":573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":191,"num_write_kb":12224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"23'235","reported_seq":259,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:55.807732+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:41:55.807732+0000","last_peered":"2024-09-11T14:41:55.807732+0000","last_clean":"2024-09-11T14:41:55.807732+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:41:55.807732+0000","last_undegraded":"2024-09-11T14:41:55.807732+0000","last_fullsized":"2024-09-11T14:41:55.807732+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15400960,"num_objects":235,"num_object_clones":0,"num_object_copies":705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":235,"num_write_kb":15040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"23'192","reported_seq":216,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:54.149382+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:41:54.149382+0000","last_peered":"2024-09-11T14:41:54.149382+0000","last_clean":"2024-09-11T14:41:54.149382+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:41:54.149382+0000","last_undegraded":"2024-09-11T14:41:54.149382+0000","last_fullsized":"2024-09-11T14:41:54.149382+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"23'215","reported_seq":239,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:54.526541+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:41:54.526541+0000","last_peered":"2024-09-11T14:41:54.526541+0000","last_clean":"2024-09-11T14:41:54.526541+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:41:54.526541+0000","last_undegraded":"2024-09-11T14:41:54.526541+0000","last_fullsized":"2024-09-11T14:41:54.526541+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14090240,"num_objects":215,"num_object_clones":0,"num_object_copies":645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":215,"num_write_kb":13760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"23'200","reported_seq":224,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:54.512453+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:41:54.512453+0000","last_peered":"2024-09-11T14:41:54.512453+0000","last_clean":"2024-09-11T14:41:54.512453+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:41:54.512453+0000","last_undegraded":"2024-09-11T14:41:54.512453+0000","last_fullsized":"2024-09-11T14:41:54.512453+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13107200,"num_objects":200,"num_object_clones":0,"num_object_copies":600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":12800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"23'219","reported_seq":243,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:54.726677+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:41:54.726677+0000","last_peered":"2024-09-11T14:41:54.726677+0000","last_clean":"2024-09-11T14:41:54.726677+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:41:54.726677+0000","last_undegraded":"2024-09-11T14:41:54.726677+0000","last_fullsized":"2024-09-11T14:41:54.726677+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14352384,"num_objects":219,"num_object_clones":0,"num_object_copies":657,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":219,"num_write_kb":14016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"23'193","reported_seq":217,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:54.607243+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:41:54.607243+0000","last_peered":"2024-09-11T14:41:54.607243+0000","last_clean":"2024-09-11T14:41:54.607243+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:41:54.607243+0000","last_undegraded":"2024-09-11T14:41:54.607243+0000","last_fullsized":"2024-09-11T14:41:54.607243+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12648448,"num_objects":193,"num_object_clones":0,"num_object_copies":579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":193,"num_write_kb":12352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"23'196","reported_seq":220,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:54.354325+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:41:54.354325+0000","last_peered":"2024-09-11T14:41:54.354325+0000","last_clean":"2024-09-11T14:41:54.354325+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:41:54.354325+0000","last_undegraded":"2024-09-11T14:41:54.354325+0000","last_fullsized":"2024-09-11T14:41:54.354325+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12845056,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":196,"num_write_kb":12544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":109,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:33.882331+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:41:33.882331+0000","last_peered":"2024-09-11T14:41:33.882331+0000","last_clean":"2024-09-11T14:41:33.882331+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:41:33.882331+0000","last_undegraded":"2024-09-11T14:41:33.882331+0000","last_fullsized":"2024-09-11T14:41:33.882331+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":38,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.165313+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:41:34.165313+0000","last_peered":"2024-09-11T14:41:34.165313+0000","last_clean":"2024-09-11T14:41:34.165313+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:41:34.165313+0000","last_undegraded":"2024-09-11T14:41:34.165313+0000","last_fullsized":"2024-09-11T14:41:34.165313+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"23'198","reported_seq":222,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:54.689026+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:41:54.689026+0000","last_peered":"2024-09-11T14:41:54.689026+0000","last_clean":"2024-09-11T14:41:54.689026+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:41:54.689026+0000","last_undegraded":"2024-09-11T14:41:54.689026+0000","last_fullsized":"2024-09-11T14:41:54.689026+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12976128,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":198,"num_write_kb":12672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.022265+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:41:34.022265+0000","last_peered":"2024-09-11T14:41:34.022265+0000","last_clean":"2024-09-11T14:41:34.022265+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:41:34.022265+0000","last_undegraded":"2024-09-11T14:41:34.022265+0000","last_fullsized":"2024-09-11T14:41:34.022265+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"23'174","reported_seq":198,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:54.394834+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:41:54.394834+0000","last_peered":"2024-09-11T14:41:54.394834+0000","last_clean":"2024-09-11T14:41:54.394834+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:41:54.394834+0000","last_undegraded":"2024-09-11T14:41:54.394834+0000","last_fullsized":"2024-09-11T14:41:54.394834+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11403264,"num_objects":174,"num_object_clones":0,"num_object_copies":522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":11136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.163617+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:41:34.163617+0000","last_peered":"2024-09-11T14:41:34.163617+0000","last_clean":"2024-09-11T14:41:34.163617+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:41:34.163617+0000","last_undegraded":"2024-09-11T14:41:34.163617+0000","last_fullsized":"2024-09-11T14:41:34.163617+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"23'208","reported_seq":232,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:53.970815+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:41:53.970815+0000","last_peered":"2024-09-11T14:41:53.970815+0000","last_clean":"2024-09-11T14:41:53.970815+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:41:53.970815+0000","last_undegraded":"2024-09-11T14:41:53.970815+0000","last_fullsized":"2024-09-11T14:41:53.970815+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13631488,"num_objects":208,"num_object_clones":0,"num_object_copies":624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":208,"num_write_kb":13312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":39,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.025108+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:41:34.025108+0000","last_peered":"2024-09-11T14:41:34.025108+0000","last_clean":"2024-09-11T14:41:34.025108+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:41:34.025108+0000","last_undegraded":"2024-09-11T14:41:34.025108+0000","last_fullsized":"2024-09-11T14:41:34.025108+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"23'195","reported_seq":219,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:55.809330+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:41:55.809330+0000","last_peered":"2024-09-11T14:41:55.809330+0000","last_clean":"2024-09-11T14:41:55.809330+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:41:55.809330+0000","last_undegraded":"2024-09-11T14:41:55.809330+0000","last_fullsized":"2024-09-11T14:41:55.809330+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12779520,"num_objects":195,"num_object_clones":0,"num_object_copies":585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":195,"num_write_kb":12480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"23'197","reported_seq":221,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:54.601519+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:41:54.601519+0000","last_peered":"2024-09-11T14:41:54.601519+0000","last_clean":"2024-09-11T14:41:54.601519+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:41:54.601519+0000","last_undegraded":"2024-09-11T14:41:54.601519+0000","last_fullsized":"2024-09-11T14:41:54.601519+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":197,"log_dups_size":0,"ondisk_log_size":197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12910592,"num_objects":197,"num_object_clones":0,"num_object_copies":591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":197,"num_write_kb":12608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.028115+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:41:34.028115+0000","last_peered":"2024-09-11T14:41:34.028115+0000","last_clean":"2024-09-11T14:41:34.028115+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:41:34.028115+0000","last_undegraded":"2024-09-11T14:41:34.028115+0000","last_fullsized":"2024-09-11T14:41:34.028115+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"23'190","reported_seq":214,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:55.476585+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:41:55.476585+0000","last_peered":"2024-09-11T14:41:55.476585+0000","last_clean":"2024-09-11T14:41:55.476585+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:41:55.476585+0000","last_undegraded":"2024-09-11T14:41:55.476585+0000","last_fullsized":"2024-09-11T14:41:55.476585+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12451840,"num_objects":190,"num_object_clones":0,"num_object_copies":570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":190,"num_write_kb":12160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:33.885106+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:41:33.885106+0000","last_peered":"2024-09-11T14:41:33.885106+0000","last_clean":"2024-09-11T14:41:33.885106+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:41:33.885106+0000","last_undegraded":"2024-09-11T14:41:33.885106+0000","last_fullsized":"2024-09-11T14:41:33.885106+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"23'208","reported_seq":232,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:54.571090+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:41:54.571090+0000","last_peered":"2024-09-11T14:41:54.571090+0000","last_clean":"2024-09-11T14:41:54.571090+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:41:54.571090+0000","last_undegraded":"2024-09-11T14:41:54.571090+0000","last_fullsized":"2024-09-11T14:41:54.571090+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13631488,"num_objects":208,"num_object_clones":0,"num_object_copies":624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":208,"num_write_kb":13312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.025417+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:41:34.025417+0000","last_peered":"2024-09-11T14:41:34.025417+0000","last_clean":"2024-09-11T14:41:34.025417+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:41:34.025417+0000","last_undegraded":"2024-09-11T14:41:34.025417+0000","last_fullsized":"2024-09-11T14:41:34.025417+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"23'208","reported_seq":232,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:55.521210+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:41:55.521210+0000","last_peered":"2024-09-11T14:41:55.521210+0000","last_clean":"2024-09-11T14:41:55.521210+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:41:55.521210+0000","last_undegraded":"2024-09-11T14:41:55.521210+0000","last_fullsized":"2024-09-11T14:41:55.521210+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13631488,"num_objects":208,"num_object_clones":0,"num_object_copies":624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":208,"num_write_kb":13312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.169641+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:41:34.169641+0000","last_peered":"2024-09-11T14:41:34.169641+0000","last_clean":"2024-09-11T14:41:34.169641+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:41:34.169641+0000","last_undegraded":"2024-09-11T14:41:34.169641+0000","last_fullsized":"2024-09-11T14:41:34.169641+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":210960384,"num_objects":3219,"num_object_clones":0,"num_object_copies":9657,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3219,"num_write_kb":206016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":212932698112,"internally_reserved":0,"allocated":1278795776,"data_stored":1278795776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3219,"ondisk_log_size":3219,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":212932829184,"internally_reserved":0,"allocated":1278664704,"data_stored":1278664704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53184630784,"internally_reserved":0,"allocated":368242688,"data_stored":368242688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":289156,"kb_used_data":289156,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52008572,"statfs":{"total":53552873472,"available":53256777728,"internally_reserved":0,"allocated":296095744,"data_stored":296095744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705713,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":347884,"kb_used_data":347884,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51949844,"statfs":{"total":53552873472,"available":53196640256,"internally_reserved":0,"allocated":356233216,"data_stored":356233216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705713,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":300068,"kb_used_data":300068,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51997660,"statfs":{"total":53552873472,"available":53245603840,"internally_reserved":0,"allocated":307269632,"data_stored":307269632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705713,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":270140,"kb_used_data":270140,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52027588,"statfs":{"total":53552873472,"available":53276250112,"internally_reserved":0,"allocated":276623360,"data_stored":276623360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53184630784,"internally_reserved":0,"allocated":368242688,"data_stored":368242688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53265756160,"internally_reserved":0,"allocated":287117312,"data_stored":287117312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53235154944,"internally_reserved":0,"allocated":317718528,"data_stored":317718528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53247287296,"internally_reserved":0,"allocated":305586176,"data_stored":305586176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53184630784,"internally_reserved":0,"allocated":368242688,"data_stored":368242688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":53265756160,"internally_reserved":0,"allocated":287117312,"data_stored":287117312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":53235154944,"internally_reserved":0,"allocated":317718528,"data_stored":317718528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":53247287296,"internally_reserved":0,"allocated":305586176,"data_stored":305586176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":53184499712,"internally_reserved":0,"allocated":368373760,"data_stored":368373760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:41:58.907 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 38 16 3627 3611 5.93855 6.1875 0.154962 0.16801 2024-09-11T14:41:59.907 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 39 16 3734 3718 5.95774 6.6875 0.113643 0.167367 2024-09-11T14:42:00.218 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:42:00.361 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:00.357+0000 7ff68f591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:00.366 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:00.364+0000 7ff68f591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:00.907 INFO:tasks.radosbench.radosbench.0.smithi181.stdout:2024-09-11T14:42:00.906982+0000 min lat: 0.0249053 max lat: 1.01958 avg lat: 0.167069 2024-09-11T14:42:00.907 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T14:42:00.907 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 40 16 3839 3823 5.97285 6.5625 0.0690655 0.167069 2024-09-11T14:42:01.907 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 41 16 3932 3916 5.96892 5.8125 0.101273 0.167112 2024-09-11T14:42:02.907 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 42 16 4029 4013 5.97113 6.0625 0.0843155 0.166909 2024-09-11T14:42:02.917 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:42:03.062 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:03.058+0000 7f0ec9093640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:03.068 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:03.067+0000 7f0ec9093640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:03.158 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:42:03.159 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:42:03.159 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:42:03.314 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:03.307+0000 7f32f38d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:03.319 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:03.318+0000 7f32f38d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:03.626 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:42:03.626 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:42:03.643 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2024-09-11T14:42:02.716624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":246415923,"num_objects":3755,"num_object_clones":0,"num_object_copies":11265,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":3832,"num_write_kb":241172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3806,"ondisk_log_size":3806,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":1414136,"kb_used_data":1414136,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":207776776,"statfs":{"total":214211493888,"available":212763418624,"internally_reserved":0,"allocated":1448075264,"data_stored":1448075264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":77398016,"num_objects":1181,"num_object_clones":0,"num_object_copies":3543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1181,"num_write_kb":75584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002726"},"pg_stats":[{"pgid":"3.f","version":"23'228","reported_seq":252,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:59.724674+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:41:59.724674+0000","last_peered":"2024-09-11T14:41:59.724674+0000","last_clean":"2024-09-11T14:41:59.724674+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:41:59.724674+0000","last_undegraded":"2024-09-11T14:41:59.724674+0000","last_fullsized":"2024-09-11T14:41:59.724674+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":228,"log_dups_size":0,"ondisk_log_size":228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14942208,"num_objects":228,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":228,"num_write_kb":14592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"23'265","reported_seq":289,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:42:00.144493+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:42:00.144493+0000","last_peered":"2024-09-11T14:42:00.144493+0000","last_clean":"2024-09-11T14:42:00.144493+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:42:00.144493+0000","last_undegraded":"2024-09-11T14:42:00.144493+0000","last_fullsized":"2024-09-11T14:42:00.144493+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":265,"log_dups_size":0,"ondisk_log_size":265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17367040,"num_objects":265,"num_object_clones":0,"num_object_copies":795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":265,"num_write_kb":16960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"23'224","reported_seq":248,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:59.351311+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:41:59.351311+0000","last_peered":"2024-09-11T14:41:59.351311+0000","last_clean":"2024-09-11T14:41:59.351311+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:41:59.351311+0000","last_undegraded":"2024-09-11T14:41:59.351311+0000","last_fullsized":"2024-09-11T14:41:59.351311+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14680064,"num_objects":224,"num_object_clones":0,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":224,"num_write_kb":14336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"23'252","reported_seq":276,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:59.476656+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:41:59.476656+0000","last_peered":"2024-09-11T14:41:59.476656+0000","last_clean":"2024-09-11T14:41:59.476656+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:41:59.476656+0000","last_undegraded":"2024-09-11T14:41:59.476656+0000","last_fullsized":"2024-09-11T14:41:59.476656+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16515072,"num_objects":252,"num_object_clones":0,"num_object_copies":756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":252,"num_write_kb":16128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"23'232","reported_seq":256,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:59.497573+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:41:59.497573+0000","last_peered":"2024-09-11T14:41:59.497573+0000","last_clean":"2024-09-11T14:41:59.497573+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:41:59.497573+0000","last_undegraded":"2024-09-11T14:41:59.497573+0000","last_fullsized":"2024-09-11T14:41:59.497573+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15204352,"num_objects":232,"num_object_clones":0,"num_object_copies":696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":232,"num_write_kb":14848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"23'252","reported_seq":276,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:59.606139+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:41:59.606139+0000","last_peered":"2024-09-11T14:41:59.606139+0000","last_clean":"2024-09-11T14:41:59.606139+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:41:59.606139+0000","last_undegraded":"2024-09-11T14:41:59.606139+0000","last_fullsized":"2024-09-11T14:41:59.606139+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16515072,"num_objects":252,"num_object_clones":0,"num_object_copies":756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":252,"num_write_kb":16128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"23'225","reported_seq":249,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:59.675893+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:41:59.675893+0000","last_peered":"2024-09-11T14:41:59.675893+0000","last_clean":"2024-09-11T14:41:59.675893+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:41:59.675893+0000","last_undegraded":"2024-09-11T14:41:59.675893+0000","last_fullsized":"2024-09-11T14:41:59.675893+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":225,"log_dups_size":0,"ondisk_log_size":225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14745600,"num_objects":225,"num_object_clones":0,"num_object_copies":675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":225,"num_write_kb":14400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"23'224","reported_seq":248,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:59.690572+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:41:59.690572+0000","last_peered":"2024-09-11T14:41:59.690572+0000","last_clean":"2024-09-11T14:41:59.690572+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:41:59.690572+0000","last_undegraded":"2024-09-11T14:41:59.690572+0000","last_fullsized":"2024-09-11T14:41:59.690572+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14680064,"num_objects":224,"num_object_clones":0,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":224,"num_write_kb":14336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":109,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:33.882331+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:41:33.882331+0000","last_peered":"2024-09-11T14:41:33.882331+0000","last_clean":"2024-09-11T14:41:33.882331+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:41:33.882331+0000","last_undegraded":"2024-09-11T14:41:33.882331+0000","last_fullsized":"2024-09-11T14:41:33.882331+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":38,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.165313+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:41:34.165313+0000","last_peered":"2024-09-11T14:41:34.165313+0000","last_clean":"2024-09-11T14:41:34.165313+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:41:34.165313+0000","last_undegraded":"2024-09-11T14:41:34.165313+0000","last_fullsized":"2024-09-11T14:41:34.165313+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"23'234","reported_seq":258,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:59.669443+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:41:59.669443+0000","last_peered":"2024-09-11T14:41:59.669443+0000","last_clean":"2024-09-11T14:41:59.669443+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:41:59.669443+0000","last_undegraded":"2024-09-11T14:41:59.669443+0000","last_fullsized":"2024-09-11T14:41:59.669443+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":234,"log_dups_size":0,"ondisk_log_size":234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15335424,"num_objects":234,"num_object_clones":0,"num_object_copies":702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":234,"num_write_kb":14976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.022265+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:41:34.022265+0000","last_peered":"2024-09-11T14:41:34.022265+0000","last_clean":"2024-09-11T14:41:34.022265+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:41:34.022265+0000","last_undegraded":"2024-09-11T14:41:34.022265+0000","last_fullsized":"2024-09-11T14:41:34.022265+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"23'209","reported_seq":233,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:59.757604+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:41:59.757604+0000","last_peered":"2024-09-11T14:41:59.757604+0000","last_clean":"2024-09-11T14:41:59.757604+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:41:59.757604+0000","last_undegraded":"2024-09-11T14:41:59.757604+0000","last_fullsized":"2024-09-11T14:41:59.757604+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13697024,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":209,"num_write_kb":13376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.163617+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:41:34.163617+0000","last_peered":"2024-09-11T14:41:34.163617+0000","last_clean":"2024-09-11T14:41:34.163617+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:41:34.163617+0000","last_undegraded":"2024-09-11T14:41:34.163617+0000","last_fullsized":"2024-09-11T14:41:34.163617+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"23'239","reported_seq":263,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:59.685030+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:41:59.685030+0000","last_peered":"2024-09-11T14:41:59.685030+0000","last_clean":"2024-09-11T14:41:59.685030+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:41:59.685030+0000","last_undegraded":"2024-09-11T14:41:59.685030+0000","last_fullsized":"2024-09-11T14:41:59.685030+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":239,"log_dups_size":0,"ondisk_log_size":239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15663104,"num_objects":239,"num_object_clones":0,"num_object_copies":717,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":239,"num_write_kb":15296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":39,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.025108+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:41:34.025108+0000","last_peered":"2024-09-11T14:41:34.025108+0000","last_clean":"2024-09-11T14:41:34.025108+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:41:34.025108+0000","last_undegraded":"2024-09-11T14:41:34.025108+0000","last_fullsized":"2024-09-11T14:41:34.025108+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"23'232","reported_seq":256,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:42:00.469174+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:42:00.469174+0000","last_peered":"2024-09-11T14:42:00.469174+0000","last_clean":"2024-09-11T14:42:00.469174+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:42:00.469174+0000","last_undegraded":"2024-09-11T14:42:00.469174+0000","last_fullsized":"2024-09-11T14:42:00.469174+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15204352,"num_objects":232,"num_object_clones":0,"num_object_copies":696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":232,"num_write_kb":14848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"23'229","reported_seq":253,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:59.558381+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:41:59.558381+0000","last_peered":"2024-09-11T14:41:59.558381+0000","last_clean":"2024-09-11T14:41:59.558381+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:41:59.558381+0000","last_undegraded":"2024-09-11T14:41:59.558381+0000","last_fullsized":"2024-09-11T14:41:59.558381+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15007744,"num_objects":229,"num_object_clones":0,"num_object_copies":687,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":229,"num_write_kb":14656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.028115+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:41:34.028115+0000","last_peered":"2024-09-11T14:41:34.028115+0000","last_clean":"2024-09-11T14:41:34.028115+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:41:34.028115+0000","last_undegraded":"2024-09-11T14:41:34.028115+0000","last_fullsized":"2024-09-11T14:41:34.028115+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"23'223","reported_seq":247,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:42:00.637917+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:42:00.637917+0000","last_peered":"2024-09-11T14:42:00.637917+0000","last_clean":"2024-09-11T14:42:00.637917+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:42:00.637917+0000","last_undegraded":"2024-09-11T14:42:00.637917+0000","last_fullsized":"2024-09-11T14:42:00.637917+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14614528,"num_objects":223,"num_object_clones":0,"num_object_copies":669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":223,"num_write_kb":14272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:33.885106+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:41:33.885106+0000","last_peered":"2024-09-11T14:41:33.885106+0000","last_clean":"2024-09-11T14:41:33.885106+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:41:33.885106+0000","last_undegraded":"2024-09-11T14:41:33.885106+0000","last_fullsized":"2024-09-11T14:41:33.885106+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"23'240","reported_seq":264,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:59.264456+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:41:59.264456+0000","last_peered":"2024-09-11T14:41:59.264456+0000","last_clean":"2024-09-11T14:41:59.264456+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:41:59.264456+0000","last_undegraded":"2024-09-11T14:41:59.264456+0000","last_fullsized":"2024-09-11T14:41:59.264456+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":240,"log_dups_size":0,"ondisk_log_size":240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15728640,"num_objects":240,"num_object_clones":0,"num_object_copies":720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":240,"num_write_kb":15360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.025417+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:41:34.025417+0000","last_peered":"2024-09-11T14:41:34.025417+0000","last_clean":"2024-09-11T14:41:34.025417+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:41:34.025417+0000","last_undegraded":"2024-09-11T14:41:34.025417+0000","last_fullsized":"2024-09-11T14:41:34.025417+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"23'243","reported_seq":267,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:42:00.555177+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:42:00.555177+0000","last_peered":"2024-09-11T14:42:00.555177+0000","last_clean":"2024-09-11T14:42:00.555177+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:42:00.555177+0000","last_undegraded":"2024-09-11T14:42:00.555177+0000","last_fullsized":"2024-09-11T14:42:00.555177+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":243,"log_dups_size":0,"ondisk_log_size":243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15925248,"num_objects":243,"num_object_clones":0,"num_object_copies":729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":243,"num_write_kb":15552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.169641+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:41:34.169641+0000","last_peered":"2024-09-11T14:41:34.169641+0000","last_clean":"2024-09-11T14:41:34.169641+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:41:34.169641+0000","last_undegraded":"2024-09-11T14:41:34.169641+0000","last_fullsized":"2024-09-11T14:41:34.169641+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":245825536,"num_objects":3751,"num_object_clones":0,"num_object_copies":11253,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3751,"num_write_kb":240064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":212722458624,"internally_reserved":0,"allocated":1489035264,"data_stored":1489035264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3751,"ondisk_log_size":3751,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":212722589696,"internally_reserved":0,"allocated":1488904192,"data_stored":1488904192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53123817472,"internally_reserved":0,"allocated":429056000,"data_stored":429056000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673014,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":335564,"kb_used_data":335564,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51962164,"statfs":{"total":53552873472,"available":53209255936,"internally_reserved":0,"allocated":343617536,"data_stored":343617536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705718,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":407636,"kb_used_data":407636,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51890092,"statfs":{"total":53552873472,"available":53135454208,"internally_reserved":0,"allocated":417419264,"data_stored":417419264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705718,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":351612,"kb_used_data":351612,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51946116,"statfs":{"total":53552873472,"available":53192822784,"internally_reserved":0,"allocated":360050688,"data_stored":360050688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705718,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":319324,"kb_used_data":319324,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51978404,"statfs":{"total":53552873472,"available":53225885696,"internally_reserved":0,"allocated":326987776,"data_stored":326987776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53123817472,"internally_reserved":0,"allocated":429056000,"data_stored":429056000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53216354304,"internally_reserved":0,"allocated":336519168,"data_stored":336519168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53182763008,"internally_reserved":0,"allocated":370110464,"data_stored":370110464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53199654912,"internally_reserved":0,"allocated":353218560,"data_stored":353218560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53123817472,"internally_reserved":0,"allocated":429056000,"data_stored":429056000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":53216354304,"internally_reserved":0,"allocated":336519168,"data_stored":336519168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":53182763008,"internally_reserved":0,"allocated":370110464,"data_stored":370110464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":53199654912,"internally_reserved":0,"allocated":353218560,"data_stored":353218560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":53123686400,"internally_reserved":0,"allocated":429187072,"data_stored":429187072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:42:03.907 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 43 16 4119 4103 5.96307 5.625 0.0628661 0.167007 2024-09-11T14:42:04.907 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 44 16 4209 4193 5.95537 5.625 0.0737802 0.167426 2024-09-11T14:42:05.357 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:42:05.517 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:05.513+0000 7f6170b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:05.521 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:05.520+0000 7f6170b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:05.907 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 45 16 4286 4270 5.92997 4.8125 0.0936327 0.1679 2024-09-11T14:42:06.907 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 46 16 4379 4363 5.9274 5.8125 0.338773 0.168011 2024-09-11T14:42:07.831 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:42:07.908 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 47 16 4470 4454 5.92229 5.6875 0.125505 0.168604 2024-09-11T14:42:07.991 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:07.987+0000 7f7207f11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:07.996 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:07.995+0000 7f7207f11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:08.645 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:42:08.645 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-11T14:42:08.787 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:08.777+0000 7f65fc791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:08.791 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:08.790+0000 7f65fc791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:08.910 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 48 16 4554 4538 5.90827 5.25 0.437817 0.168683 2024-09-11T14:42:09.908 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 49 16 4647 4631 5.9063 5.8125 0.217113 0.168893 2024-09-11T14:42:09.920 INFO:teuthology.orchestra.run.smithi181.stderr:reweighted osd.0 to 1 (10000) 2024-09-11T14:42:09.945 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-11T14:42:10.102 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:10.092+0000 7f4d5af91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:10.108 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:10.107+0000 7f4d5af91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:10.416 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:42:10.580 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:10.576+0000 7f2843111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:10.584 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:10.583+0000 7f2843111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:10.908 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 50 16 4735 4719 5.89816 5.5 0.112874 0.1691 2024-09-11T14:42:10.932 INFO:teuthology.orchestra.run.smithi181.stderr:reweighted osd.1 to 1 (10000) 2024-09-11T14:42:10.959 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-11T14:42:11.180 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:11.168+0000 7faeb30d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:11.186 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:11.184+0000 7faeb30d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:11.908 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 51 16 4839 4823 5.90995 6.5 0.123984 0.16887 2024-09-11T14:42:11.933 INFO:teuthology.orchestra.run.smithi181.stderr:reweighted osd.2 to 1 (10000) 2024-09-11T14:42:11.960 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-11T14:42:12.105 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:12.093+0000 7fb7cd511640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:12.113 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:12.111+0000 7fb7cd511640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:12.908 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 52 16 4938 4922 5.91528 6.1875 0.118106 0.168795 2024-09-11T14:42:12.917 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:42:12.937 INFO:teuthology.orchestra.run.smithi181.stderr:reweighted osd.3 to 1 (10000) 2024-09-11T14:42:12.958 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-11T14:42:12.958 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:42:13.065 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:13.062+0000 7fb5b4ed1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:13.072 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:13.069+0000 7fb5b4ed1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:13.179 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:13.171+0000 7f2aec551640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:13.184 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:13.183+0000 7f2aec551640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:13.547 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:42:13.547 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:42:13.570 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":54,"stamp":"2024-09-11T14:42:12.940132+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":306446899,"num_objects":4671,"num_object_clones":0,"num_object_copies":14013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":4748,"num_write_kb":299796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4722,"ondisk_log_size":4722,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":1775580,"kb_used_data":1775580,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":207415332,"statfs":{"total":214211493888,"available":212393299968,"internally_reserved":0,"allocated":1818193920,"data_stored":1818193920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28573696,"num_objects":436,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":436,"num_write_kb":27904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.222165"},"pg_stats":[{"pgid":"3.f","version":"23'271","reported_seq":295,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:42:09.702950+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:42:09.702950+0000","last_peered":"2024-09-11T14:42:09.702950+0000","last_clean":"2024-09-11T14:42:09.702950+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:42:09.702950+0000","last_undegraded":"2024-09-11T14:42:09.702950+0000","last_fullsized":"2024-09-11T14:42:09.702950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17760256,"num_objects":271,"num_object_clones":0,"num_object_copies":813,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":271,"num_write_kb":17344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"24'335","reported_seq":362,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-11T14:42:10.401677+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:42:10.401677+0000","last_peered":"2024-09-11T14:42:10.401677+0000","last_clean":"2024-09-11T14:42:10.401677+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:42:10.401677+0000","last_undegraded":"2024-09-11T14:42:10.401677+0000","last_fullsized":"2024-09-11T14:42:10.401677+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21954560,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":335,"num_write_kb":21440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"23'277","reported_seq":301,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:42:09.731743+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:42:09.731743+0000","last_peered":"2024-09-11T14:42:09.731743+0000","last_clean":"2024-09-11T14:42:09.731743+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:42:09.731743+0000","last_undegraded":"2024-09-11T14:42:09.731743+0000","last_fullsized":"2024-09-11T14:42:09.731743+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":277,"log_dups_size":0,"ondisk_log_size":277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18153472,"num_objects":277,"num_object_clones":0,"num_object_copies":831,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":277,"num_write_kb":17728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"23'315","reported_seq":339,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:42:09.260628+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:42:09.260628+0000","last_peered":"2024-09-11T14:42:09.260628+0000","last_clean":"2024-09-11T14:42:09.260628+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:42:09.260628+0000","last_undegraded":"2024-09-11T14:42:09.260628+0000","last_fullsized":"2024-09-11T14:42:09.260628+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20643840,"num_objects":315,"num_object_clones":0,"num_object_copies":945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":315,"num_write_kb":20160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"23'284","reported_seq":308,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:42:09.783667+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:42:09.783667+0000","last_peered":"2024-09-11T14:42:09.783667+0000","last_clean":"2024-09-11T14:42:09.783667+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:42:09.783667+0000","last_undegraded":"2024-09-11T14:42:09.783667+0000","last_fullsized":"2024-09-11T14:42:09.783667+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":284,"log_dups_size":0,"ondisk_log_size":284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18612224,"num_objects":284,"num_object_clones":0,"num_object_copies":852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":284,"num_write_kb":18176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"23'325","reported_seq":349,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:42:09.637590+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:42:09.637590+0000","last_peered":"2024-09-11T14:42:09.637590+0000","last_clean":"2024-09-11T14:42:09.637590+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:42:09.637590+0000","last_undegraded":"2024-09-11T14:42:09.637590+0000","last_fullsized":"2024-09-11T14:42:09.637590+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21299200,"num_objects":325,"num_object_clones":0,"num_object_copies":975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":325,"num_write_kb":20800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"23'280","reported_seq":304,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:42:09.807129+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:42:09.807129+0000","last_peered":"2024-09-11T14:42:09.807129+0000","last_clean":"2024-09-11T14:42:09.807129+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:42:09.807129+0000","last_undegraded":"2024-09-11T14:42:09.807129+0000","last_fullsized":"2024-09-11T14:42:09.807129+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":280,"log_dups_size":0,"ondisk_log_size":280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18350080,"num_objects":280,"num_object_clones":0,"num_object_copies":840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":280,"num_write_kb":17920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"23'258","reported_seq":282,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:42:09.171797+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:42:09.171797+0000","last_peered":"2024-09-11T14:42:09.171797+0000","last_clean":"2024-09-11T14:42:09.171797+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:42:09.171797+0000","last_undegraded":"2024-09-11T14:42:09.171797+0000","last_fullsized":"2024-09-11T14:42:09.171797+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16908288,"num_objects":258,"num_object_clones":0,"num_object_copies":774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":258,"num_write_kb":16512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":109,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:33.882331+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:41:33.882331+0000","last_peered":"2024-09-11T14:41:33.882331+0000","last_clean":"2024-09-11T14:41:33.882331+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:41:33.882331+0000","last_undegraded":"2024-09-11T14:41:33.882331+0000","last_fullsized":"2024-09-11T14:41:33.882331+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":38,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.165313+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:41:34.165313+0000","last_peered":"2024-09-11T14:41:34.165313+0000","last_clean":"2024-09-11T14:41:34.165313+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:41:34.165313+0000","last_undegraded":"2024-09-11T14:41:34.165313+0000","last_fullsized":"2024-09-11T14:41:34.165313+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"23'287","reported_seq":311,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:42:09.423359+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:42:09.423359+0000","last_peered":"2024-09-11T14:42:09.423359+0000","last_clean":"2024-09-11T14:42:09.423359+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:42:09.423359+0000","last_undegraded":"2024-09-11T14:42:09.423359+0000","last_fullsized":"2024-09-11T14:42:09.423359+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18808832,"num_objects":287,"num_object_clones":0,"num_object_copies":861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":287,"num_write_kb":18368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.022265+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:41:34.022265+0000","last_peered":"2024-09-11T14:41:34.022265+0000","last_clean":"2024-09-11T14:41:34.022265+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:41:34.022265+0000","last_undegraded":"2024-09-11T14:41:34.022265+0000","last_fullsized":"2024-09-11T14:41:34.022265+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"23'259","reported_seq":283,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:42:09.624221+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:42:09.624221+0000","last_peered":"2024-09-11T14:42:09.624221+0000","last_clean":"2024-09-11T14:42:09.624221+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:42:09.624221+0000","last_undegraded":"2024-09-11T14:42:09.624221+0000","last_fullsized":"2024-09-11T14:42:09.624221+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16973824,"num_objects":259,"num_object_clones":0,"num_object_copies":777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":259,"num_write_kb":16576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.163617+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:41:34.163617+0000","last_peered":"2024-09-11T14:41:34.163617+0000","last_clean":"2024-09-11T14:41:34.163617+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:41:34.163617+0000","last_undegraded":"2024-09-11T14:41:34.163617+0000","last_fullsized":"2024-09-11T14:41:34.163617+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"23'303","reported_seq":327,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:42:09.557264+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:42:09.557264+0000","last_peered":"2024-09-11T14:42:09.557264+0000","last_clean":"2024-09-11T14:42:09.557264+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:42:09.557264+0000","last_undegraded":"2024-09-11T14:42:09.557264+0000","last_fullsized":"2024-09-11T14:42:09.557264+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19857408,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":303,"num_write_kb":19392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":39,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.025108+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:41:34.025108+0000","last_peered":"2024-09-11T14:41:34.025108+0000","last_clean":"2024-09-11T14:41:34.025108+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:41:34.025108+0000","last_undegraded":"2024-09-11T14:41:34.025108+0000","last_fullsized":"2024-09-11T14:41:34.025108+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"24'287","reported_seq":314,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-11T14:42:10.369980+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:42:10.369980+0000","last_peered":"2024-09-11T14:42:10.369980+0000","last_clean":"2024-09-11T14:42:10.369980+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:42:10.369980+0000","last_undegraded":"2024-09-11T14:42:10.369980+0000","last_fullsized":"2024-09-11T14:42:10.369980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18808832,"num_objects":287,"num_object_clones":0,"num_object_copies":861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":287,"num_write_kb":18368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"23'279","reported_seq":303,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:42:09.038655+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:42:09.038655+0000","last_peered":"2024-09-11T14:42:09.038655+0000","last_clean":"2024-09-11T14:42:09.038655+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:42:09.038655+0000","last_undegraded":"2024-09-11T14:42:09.038655+0000","last_fullsized":"2024-09-11T14:42:09.038655+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18284544,"num_objects":279,"num_object_clones":0,"num_object_copies":837,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":279,"num_write_kb":17856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":40,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-11T14:42:09.988557+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:42:09.988557+0000","last_peered":"2024-09-11T14:42:09.988557+0000","last_clean":"2024-09-11T14:42:09.988557+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:42:09.988557+0000","last_undegraded":"2024-09-11T14:42:09.988557+0000","last_fullsized":"2024-09-11T14:42:09.988557+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"24'291","reported_seq":318,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-11T14:42:10.589479+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:42:10.589479+0000","last_peered":"2024-09-11T14:42:10.589479+0000","last_clean":"2024-09-11T14:42:10.589479+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:42:10.589479+0000","last_undegraded":"2024-09-11T14:42:10.589479+0000","last_fullsized":"2024-09-11T14:42:10.589479+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19070976,"num_objects":291,"num_object_clones":0,"num_object_copies":873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":291,"num_write_kb":18624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:33.885106+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:41:33.885106+0000","last_peered":"2024-09-11T14:41:33.885106+0000","last_clean":"2024-09-11T14:41:33.885106+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:41:33.885106+0000","last_undegraded":"2024-09-11T14:41:33.885106+0000","last_fullsized":"2024-09-11T14:41:33.885106+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"23'306","reported_seq":330,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:42:09.445620+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:42:09.445620+0000","last_peered":"2024-09-11T14:42:09.445620+0000","last_clean":"2024-09-11T14:42:09.445620+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:42:09.445620+0000","last_undegraded":"2024-09-11T14:42:09.445620+0000","last_fullsized":"2024-09-11T14:42:09.445620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":306,"log_dups_size":0,"ondisk_log_size":306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20054016,"num_objects":306,"num_object_clones":0,"num_object_copies":918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":306,"num_write_kb":19584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":40,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-11T14:42:09.985194+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:42:09.985194+0000","last_peered":"2024-09-11T14:42:09.985194+0000","last_clean":"2024-09-11T14:42:09.985194+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:42:09.985194+0000","last_undegraded":"2024-09-11T14:42:09.985194+0000","last_fullsized":"2024-09-11T14:42:09.985194+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"24'310","reported_seq":337,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-11T14:42:10.629066+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:42:10.629066+0000","last_peered":"2024-09-11T14:42:10.629066+0000","last_clean":"2024-09-11T14:42:10.629066+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:42:10.629066+0000","last_undegraded":"2024-09-11T14:42:10.629066+0000","last_fullsized":"2024-09-11T14:42:10.629066+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20316160,"num_objects":310,"num_object_clones":0,"num_object_copies":930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":310,"num_write_kb":19840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.169641+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:41:34.169641+0000","last_peered":"2024-09-11T14:41:34.169641+0000","last_clean":"2024-09-11T14:41:34.169641+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:41:34.169641+0000","last_undegraded":"2024-09-11T14:41:34.169641+0000","last_fullsized":"2024-09-11T14:41:34.169641+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":305856512,"num_objects":4667,"num_object_clones":0,"num_object_copies":14001,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4667,"num_write_kb":298688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":212359131136,"internally_reserved":0,"allocated":1852362752,"data_stored":1852362752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4667,"ondisk_log_size":4667,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":212359262208,"internally_reserved":0,"allocated":1852231680,"data_stored":1852231680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53018771456,"internally_reserved":0,"allocated":534102016,"data_stored":534102016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":419716,"kb_used_data":419716,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51878012,"statfs":{"total":53552873472,"available":53123084288,"internally_reserved":0,"allocated":429789184,"data_stored":429789184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705728,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":513188,"kb_used_data":513188,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51784540,"statfs":{"total":53552873472,"available":53027368960,"internally_reserved":0,"allocated":525504512,"data_stored":525504512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705728,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":441012,"kb_used_data":441012,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51856716,"statfs":{"total":53552873472,"available":53101277184,"internally_reserved":0,"allocated":451596288,"data_stored":451596288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705728,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":401664,"kb_used_data":401664,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51896064,"statfs":{"total":53552873472,"available":53141569536,"internally_reserved":0,"allocated":411303936,"data_stored":411303936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53018771456,"internally_reserved":0,"allocated":534102016,"data_stored":534102016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53132881920,"internally_reserved":0,"allocated":419991552,"data_stored":419991552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53093076992,"internally_reserved":0,"allocated":459796480,"data_stored":459796480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53114531840,"internally_reserved":0,"allocated":438341632,"data_stored":438341632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53018771456,"internally_reserved":0,"allocated":534102016,"data_stored":534102016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":53132881920,"internally_reserved":0,"allocated":419991552,"data_stored":419991552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":53093076992,"internally_reserved":0,"allocated":459796480,"data_stored":459796480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":53114531840,"internally_reserved":0,"allocated":438341632,"data_stored":438341632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":53018640384,"internally_reserved":0,"allocated":534233088,"data_stored":534233088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:42:13.572 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:42:13.746 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:13.736+0000 7f7c8b4fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:13.753 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:13.752+0000 7f7c8b4fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:13.908 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 53 16 5036 5020 5.91922 6.125 0.214714 0.168645 2024-09-11T14:42:14.041 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:42:14.042 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:42:14.067 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":54,"stamp":"2024-09-11T14:42:12.940132+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":306446899,"num_objects":4671,"num_object_clones":0,"num_object_copies":14013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":4748,"num_write_kb":299796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4722,"ondisk_log_size":4722,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":1775580,"kb_used_data":1775580,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":207415332,"statfs":{"total":214211493888,"available":212393299968,"internally_reserved":0,"allocated":1818193920,"data_stored":1818193920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28573696,"num_objects":436,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":436,"num_write_kb":27904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.222165"},"pg_stats":[{"pgid":"3.f","version":"23'271","reported_seq":295,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:42:09.702950+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:42:09.702950+0000","last_peered":"2024-09-11T14:42:09.702950+0000","last_clean":"2024-09-11T14:42:09.702950+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:42:09.702950+0000","last_undegraded":"2024-09-11T14:42:09.702950+0000","last_fullsized":"2024-09-11T14:42:09.702950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17760256,"num_objects":271,"num_object_clones":0,"num_object_copies":813,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":271,"num_write_kb":17344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"24'335","reported_seq":362,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-11T14:42:10.401677+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:42:10.401677+0000","last_peered":"2024-09-11T14:42:10.401677+0000","last_clean":"2024-09-11T14:42:10.401677+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:42:10.401677+0000","last_undegraded":"2024-09-11T14:42:10.401677+0000","last_fullsized":"2024-09-11T14:42:10.401677+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21954560,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":335,"num_write_kb":21440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"23'277","reported_seq":301,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:42:09.731743+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:42:09.731743+0000","last_peered":"2024-09-11T14:42:09.731743+0000","last_clean":"2024-09-11T14:42:09.731743+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:42:09.731743+0000","last_undegraded":"2024-09-11T14:42:09.731743+0000","last_fullsized":"2024-09-11T14:42:09.731743+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":277,"log_dups_size":0,"ondisk_log_size":277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18153472,"num_objects":277,"num_object_clones":0,"num_object_copies":831,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":277,"num_write_kb":17728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"23'315","reported_seq":339,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:42:09.260628+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:42:09.260628+0000","last_peered":"2024-09-11T14:42:09.260628+0000","last_clean":"2024-09-11T14:42:09.260628+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:42:09.260628+0000","last_undegraded":"2024-09-11T14:42:09.260628+0000","last_fullsized":"2024-09-11T14:42:09.260628+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20643840,"num_objects":315,"num_object_clones":0,"num_object_copies":945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":315,"num_write_kb":20160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"23'284","reported_seq":308,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:42:09.783667+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:42:09.783667+0000","last_peered":"2024-09-11T14:42:09.783667+0000","last_clean":"2024-09-11T14:42:09.783667+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:42:09.783667+0000","last_undegraded":"2024-09-11T14:42:09.783667+0000","last_fullsized":"2024-09-11T14:42:09.783667+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":284,"log_dups_size":0,"ondisk_log_size":284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18612224,"num_objects":284,"num_object_clones":0,"num_object_copies":852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":284,"num_write_kb":18176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"23'325","reported_seq":349,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:42:09.637590+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:42:09.637590+0000","last_peered":"2024-09-11T14:42:09.637590+0000","last_clean":"2024-09-11T14:42:09.637590+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:42:09.637590+0000","last_undegraded":"2024-09-11T14:42:09.637590+0000","last_fullsized":"2024-09-11T14:42:09.637590+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21299200,"num_objects":325,"num_object_clones":0,"num_object_copies":975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":325,"num_write_kb":20800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"23'280","reported_seq":304,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:42:09.807129+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:42:09.807129+0000","last_peered":"2024-09-11T14:42:09.807129+0000","last_clean":"2024-09-11T14:42:09.807129+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:42:09.807129+0000","last_undegraded":"2024-09-11T14:42:09.807129+0000","last_fullsized":"2024-09-11T14:42:09.807129+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":280,"log_dups_size":0,"ondisk_log_size":280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18350080,"num_objects":280,"num_object_clones":0,"num_object_copies":840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":280,"num_write_kb":17920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"23'258","reported_seq":282,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:42:09.171797+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:42:09.171797+0000","last_peered":"2024-09-11T14:42:09.171797+0000","last_clean":"2024-09-11T14:42:09.171797+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:42:09.171797+0000","last_undegraded":"2024-09-11T14:42:09.171797+0000","last_fullsized":"2024-09-11T14:42:09.171797+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16908288,"num_objects":258,"num_object_clones":0,"num_object_copies":774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":258,"num_write_kb":16512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":109,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:33.882331+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:41:33.882331+0000","last_peered":"2024-09-11T14:41:33.882331+0000","last_clean":"2024-09-11T14:41:33.882331+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:41:33.882331+0000","last_undegraded":"2024-09-11T14:41:33.882331+0000","last_fullsized":"2024-09-11T14:41:33.882331+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":38,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.165313+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:41:34.165313+0000","last_peered":"2024-09-11T14:41:34.165313+0000","last_clean":"2024-09-11T14:41:34.165313+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:41:34.165313+0000","last_undegraded":"2024-09-11T14:41:34.165313+0000","last_fullsized":"2024-09-11T14:41:34.165313+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"23'287","reported_seq":311,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:42:09.423359+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:42:09.423359+0000","last_peered":"2024-09-11T14:42:09.423359+0000","last_clean":"2024-09-11T14:42:09.423359+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:42:09.423359+0000","last_undegraded":"2024-09-11T14:42:09.423359+0000","last_fullsized":"2024-09-11T14:42:09.423359+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18808832,"num_objects":287,"num_object_clones":0,"num_object_copies":861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":287,"num_write_kb":18368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.022265+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:41:34.022265+0000","last_peered":"2024-09-11T14:41:34.022265+0000","last_clean":"2024-09-11T14:41:34.022265+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:41:34.022265+0000","last_undegraded":"2024-09-11T14:41:34.022265+0000","last_fullsized":"2024-09-11T14:41:34.022265+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"23'259","reported_seq":283,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:42:09.624221+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:42:09.624221+0000","last_peered":"2024-09-11T14:42:09.624221+0000","last_clean":"2024-09-11T14:42:09.624221+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:42:09.624221+0000","last_undegraded":"2024-09-11T14:42:09.624221+0000","last_fullsized":"2024-09-11T14:42:09.624221+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16973824,"num_objects":259,"num_object_clones":0,"num_object_copies":777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":259,"num_write_kb":16576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.163617+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:41:34.163617+0000","last_peered":"2024-09-11T14:41:34.163617+0000","last_clean":"2024-09-11T14:41:34.163617+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:41:34.163617+0000","last_undegraded":"2024-09-11T14:41:34.163617+0000","last_fullsized":"2024-09-11T14:41:34.163617+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"23'303","reported_seq":327,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:42:09.557264+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:42:09.557264+0000","last_peered":"2024-09-11T14:42:09.557264+0000","last_clean":"2024-09-11T14:42:09.557264+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:42:09.557264+0000","last_undegraded":"2024-09-11T14:42:09.557264+0000","last_fullsized":"2024-09-11T14:42:09.557264+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19857408,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":303,"num_write_kb":19392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":39,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.025108+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:41:34.025108+0000","last_peered":"2024-09-11T14:41:34.025108+0000","last_clean":"2024-09-11T14:41:34.025108+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:41:34.025108+0000","last_undegraded":"2024-09-11T14:41:34.025108+0000","last_fullsized":"2024-09-11T14:41:34.025108+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"24'287","reported_seq":314,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-11T14:42:10.369980+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:42:10.369980+0000","last_peered":"2024-09-11T14:42:10.369980+0000","last_clean":"2024-09-11T14:42:10.369980+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:42:10.369980+0000","last_undegraded":"2024-09-11T14:42:10.369980+0000","last_fullsized":"2024-09-11T14:42:10.369980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18808832,"num_objects":287,"num_object_clones":0,"num_object_copies":861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":287,"num_write_kb":18368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"23'279","reported_seq":303,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:42:09.038655+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:42:09.038655+0000","last_peered":"2024-09-11T14:42:09.038655+0000","last_clean":"2024-09-11T14:42:09.038655+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:42:09.038655+0000","last_undegraded":"2024-09-11T14:42:09.038655+0000","last_fullsized":"2024-09-11T14:42:09.038655+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18284544,"num_objects":279,"num_object_clones":0,"num_object_copies":837,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":279,"num_write_kb":17856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":40,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-11T14:42:09.988557+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:42:09.988557+0000","last_peered":"2024-09-11T14:42:09.988557+0000","last_clean":"2024-09-11T14:42:09.988557+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:42:09.988557+0000","last_undegraded":"2024-09-11T14:42:09.988557+0000","last_fullsized":"2024-09-11T14:42:09.988557+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"24'291","reported_seq":318,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-11T14:42:10.589479+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:42:10.589479+0000","last_peered":"2024-09-11T14:42:10.589479+0000","last_clean":"2024-09-11T14:42:10.589479+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:42:10.589479+0000","last_undegraded":"2024-09-11T14:42:10.589479+0000","last_fullsized":"2024-09-11T14:42:10.589479+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19070976,"num_objects":291,"num_object_clones":0,"num_object_copies":873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":291,"num_write_kb":18624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:33.885106+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:41:33.885106+0000","last_peered":"2024-09-11T14:41:33.885106+0000","last_clean":"2024-09-11T14:41:33.885106+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:41:33.885106+0000","last_undegraded":"2024-09-11T14:41:33.885106+0000","last_fullsized":"2024-09-11T14:41:33.885106+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"23'306","reported_seq":330,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:42:09.445620+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:42:09.445620+0000","last_peered":"2024-09-11T14:42:09.445620+0000","last_clean":"2024-09-11T14:42:09.445620+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:42:09.445620+0000","last_undegraded":"2024-09-11T14:42:09.445620+0000","last_fullsized":"2024-09-11T14:42:09.445620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":306,"log_dups_size":0,"ondisk_log_size":306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20054016,"num_objects":306,"num_object_clones":0,"num_object_copies":918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":306,"num_write_kb":19584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":40,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-11T14:42:09.985194+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:42:09.985194+0000","last_peered":"2024-09-11T14:42:09.985194+0000","last_clean":"2024-09-11T14:42:09.985194+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:42:09.985194+0000","last_undegraded":"2024-09-11T14:42:09.985194+0000","last_fullsized":"2024-09-11T14:42:09.985194+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"24'310","reported_seq":337,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-11T14:42:10.629066+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:42:10.629066+0000","last_peered":"2024-09-11T14:42:10.629066+0000","last_clean":"2024-09-11T14:42:10.629066+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:42:10.629066+0000","last_undegraded":"2024-09-11T14:42:10.629066+0000","last_fullsized":"2024-09-11T14:42:10.629066+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20316160,"num_objects":310,"num_object_clones":0,"num_object_copies":930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":310,"num_write_kb":19840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T14:41:34.169641+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:41:34.169641+0000","last_peered":"2024-09-11T14:41:34.169641+0000","last_clean":"2024-09-11T14:41:34.169641+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:41:34.169641+0000","last_undegraded":"2024-09-11T14:41:34.169641+0000","last_fullsized":"2024-09-11T14:41:34.169641+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":305856512,"num_objects":4667,"num_object_clones":0,"num_object_copies":14001,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4667,"num_write_kb":298688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":212359131136,"internally_reserved":0,"allocated":1852362752,"data_stored":1852362752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4667,"ondisk_log_size":4667,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":212359262208,"internally_reserved":0,"allocated":1852231680,"data_stored":1852231680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53018771456,"internally_reserved":0,"allocated":534102016,"data_stored":534102016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":419716,"kb_used_data":419716,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51878012,"statfs":{"total":53552873472,"available":53123084288,"internally_reserved":0,"allocated":429789184,"data_stored":429789184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705728,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":513188,"kb_used_data":513188,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51784540,"statfs":{"total":53552873472,"available":53027368960,"internally_reserved":0,"allocated":525504512,"data_stored":525504512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705728,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":441012,"kb_used_data":441012,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51856716,"statfs":{"total":53552873472,"available":53101277184,"internally_reserved":0,"allocated":451596288,"data_stored":451596288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705728,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":401664,"kb_used_data":401664,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51896064,"statfs":{"total":53552873472,"available":53141569536,"internally_reserved":0,"allocated":411303936,"data_stored":411303936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53018771456,"internally_reserved":0,"allocated":534102016,"data_stored":534102016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53132881920,"internally_reserved":0,"allocated":419991552,"data_stored":419991552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53093076992,"internally_reserved":0,"allocated":459796480,"data_stored":459796480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53114531840,"internally_reserved":0,"allocated":438341632,"data_stored":438341632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53018771456,"internally_reserved":0,"allocated":534102016,"data_stored":534102016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":53132881920,"internally_reserved":0,"allocated":419991552,"data_stored":419991552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":53093076992,"internally_reserved":0,"allocated":459796480,"data_stored":459796480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":53114531840,"internally_reserved":0,"allocated":438341632,"data_stored":438341632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":53018640384,"internally_reserved":0,"allocated":534233088,"data_stored":534233088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:42:14.068 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-11T14:42:14.068 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:42:14.068 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-11T14:42:14.069 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-11T14:42:14.242 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:14.233+0000 7ff9c9151640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:14.251 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:14.250+0000 7ff9c9151640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:14.346 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:42:14.347 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:42:14.347 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:42:14.347 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:42:14.347 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:42:14.347 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:42:14.347 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:42:14.347 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:42:14.347 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:42:14.347 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:42:14.347 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:42:14.347 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:42:14.351 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:42:14.351 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:42:14.351 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-11T14:42:14.503 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:14.495+0000 7f2f606c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:14.509 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:14.508+0000 7f2f606c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:14.634 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:42:14.634 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:42:14.634 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:42:14.634 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:42:14.634 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:42:14.634 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:42:14.634 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:42:14.635 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:42:14.635 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:42:14.635 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:42:14.635 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:42:14.635 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:42:14.638 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:42:14.639 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:42:14.639 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-11T14:42:14.786 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:14.775+0000 7f91ec6fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:14.793 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:14.791+0000 7f91ec6fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:14.817 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:42:14.817 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:42:14.817 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:42:14.817 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:42:14.817 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:42:14.818 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:42:14.818 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:42:14.818 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:42:14.818 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:42:14.818 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:42:14.818 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:42:14.818 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:42:14.821 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:42:14.821 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:42:14.821 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-11T14:42:14.908 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 54 16 5126 5110 5.91376 5.625 0.396796 0.168756 2024-09-11T14:42:14.961 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:14.948+0000 7f05a1511640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:14.970 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:14.968+0000 7f05a1511640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:15.129 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:42:15.129 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:42:15.129 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:42:15.129 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:42:15.129 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:42:15.129 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:42:15.129 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:42:15.130 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:42:15.130 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:42:15.130 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:42:15.130 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:42:15.130 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:42:15.134 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:42:15.134 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:42:15.631 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:42:15.806 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:15.803+0000 7f8942591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:15.811 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:15.810+0000 7f8942591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:15.909 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 55 16 5232 5216 5.92668 6.625 0.126705 0.168482 2024-09-11T14:42:16.909 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 56 16 5329 5313 5.92909 6.0625 0.0754198 0.168339 2024-09-11T14:42:17.909 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 57 16 5434 5418 5.94019 6.5625 0.0978915 0.168153 2024-09-11T14:42:18.129 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:42:18.276 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:18.271+0000 7f0ea9e7d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:18.282 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:18.280+0000 7f0ea9e7d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:18.909 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 58 16 5528 5512 5.93905 5.875 0.0804859 0.168019 2024-09-11T14:42:19.909 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 59 16 5629 5613 5.94537 6.3125 0.27633 0.16778 2024-09-11T14:42:20.135 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:42:20.135 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:42:20.136 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:42:20.273 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:20.266+0000 7f33700fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:20.277 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:20.276+0000 7f33700fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:20.600 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:42:20.600 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:42:20.615 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":57,"stamp":"2024-09-11T14:42:18.719997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":337445427,"num_objects":5144,"num_object_clones":0,"num_object_copies":15432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":5221,"num_write_kb":330068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5195,"ondisk_log_size":5195,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":1960532,"kb_used_data":1960532,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":207230380,"statfs":{"total":214211493888,"available":212203909120,"internally_reserved":0,"allocated":2007584768,"data_stored":2007584768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30998528,"num_objects":473,"num_object_clones":0,"num_object_copies":1419,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":473,"num_write_kb":30272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.785182"},"pg_stats":[{"pgid":"3.f","version":"27'292","reported_seq":328,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:14.750552+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:42:14.750552+0000","last_peered":"2024-09-11T14:42:14.750552+0000","last_clean":"2024-09-11T14:42:14.750552+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:42:14.750552+0000","last_undegraded":"2024-09-11T14:42:14.750552+0000","last_fullsized":"2024-09-11T14:42:14.750552+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":292,"log_dups_size":0,"ondisk_log_size":292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19136512,"num_objects":292,"num_object_clones":0,"num_object_copies":876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":292,"num_write_kb":18688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"27'370","reported_seq":406,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:15.151446+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:42:15.151446+0000","last_peered":"2024-09-11T14:42:15.151446+0000","last_clean":"2024-09-11T14:42:15.151446+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:42:15.151446+0000","last_undegraded":"2024-09-11T14:42:15.151446+0000","last_fullsized":"2024-09-11T14:42:15.151446+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24248320,"num_objects":370,"num_object_clones":0,"num_object_copies":1110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":370,"num_write_kb":23680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"27'303","reported_seq":339,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:14.504058+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:42:14.504058+0000","last_peered":"2024-09-11T14:42:14.504058+0000","last_clean":"2024-09-11T14:42:14.504058+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:42:14.504058+0000","last_undegraded":"2024-09-11T14:42:14.504058+0000","last_fullsized":"2024-09-11T14:42:14.504058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19857408,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":303,"num_write_kb":19392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"27'340","reported_seq":376,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:14.531503+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:42:14.531503+0000","last_peered":"2024-09-11T14:42:14.531503+0000","last_clean":"2024-09-11T14:42:14.531503+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:42:14.531503+0000","last_undegraded":"2024-09-11T14:42:14.531503+0000","last_fullsized":"2024-09-11T14:42:14.531503+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":340,"log_dups_size":0,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22282240,"num_objects":340,"num_object_clones":0,"num_object_copies":1020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":340,"num_write_kb":21760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"27'314","reported_seq":350,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:14.628450+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:42:14.628450+0000","last_peered":"2024-09-11T14:42:14.628450+0000","last_clean":"2024-09-11T14:42:14.628450+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:42:14.628450+0000","last_undegraded":"2024-09-11T14:42:14.628450+0000","last_fullsized":"2024-09-11T14:42:14.628450+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20578304,"num_objects":314,"num_object_clones":0,"num_object_copies":942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":314,"num_write_kb":20096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"27'359","reported_seq":395,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:14.755175+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:42:14.755175+0000","last_peered":"2024-09-11T14:42:14.755175+0000","last_clean":"2024-09-11T14:42:14.755175+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:42:14.755175+0000","last_undegraded":"2024-09-11T14:42:14.755175+0000","last_fullsized":"2024-09-11T14:42:14.755175+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23527424,"num_objects":359,"num_object_clones":0,"num_object_copies":1077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":359,"num_write_kb":22976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"27'324","reported_seq":360,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:14.686645+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:42:14.686645+0000","last_peered":"2024-09-11T14:42:14.686645+0000","last_clean":"2024-09-11T14:42:14.686645+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:42:14.686645+0000","last_undegraded":"2024-09-11T14:42:14.686645+0000","last_fullsized":"2024-09-11T14:42:14.686645+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21233664,"num_objects":324,"num_object_clones":0,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":324,"num_write_kb":20736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"27'287","reported_seq":323,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:14.086489+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:42:14.086489+0000","last_peered":"2024-09-11T14:42:14.086489+0000","last_clean":"2024-09-11T14:42:14.086489+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:42:14.086489+0000","last_undegraded":"2024-09-11T14:42:14.086489+0000","last_fullsized":"2024-09-11T14:42:14.086489+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18808832,"num_objects":287,"num_object_clones":0,"num_object_copies":861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":287,"num_write_kb":18368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":121,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084543+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:42:13.084543+0000","last_peered":"2024-09-11T14:42:13.084543+0000","last_clean":"2024-09-11T14:42:13.084543+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:42:13.084543+0000","last_undegraded":"2024-09-11T14:42:13.084543+0000","last_fullsized":"2024-09-11T14:42:13.084543+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":50,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.184283+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:42:13.184283+0000","last_peered":"2024-09-11T14:42:13.184283+0000","last_clean":"2024-09-11T14:42:13.184283+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:42:13.184283+0000","last_undegraded":"2024-09-11T14:42:13.184283+0000","last_fullsized":"2024-09-11T14:42:13.184283+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"27'323","reported_seq":359,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:14.410456+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:42:14.410456+0000","last_peered":"2024-09-11T14:42:14.410456+0000","last_clean":"2024-09-11T14:42:14.410456+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:42:14.410456+0000","last_undegraded":"2024-09-11T14:42:14.410456+0000","last_fullsized":"2024-09-11T14:42:14.410456+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21168128,"num_objects":323,"num_object_clones":0,"num_object_copies":969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":323,"num_write_kb":20672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084333+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:42:13.084333+0000","last_peered":"2024-09-11T14:42:13.084333+0000","last_clean":"2024-09-11T14:42:13.084333+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:42:13.084333+0000","last_undegraded":"2024-09-11T14:42:13.084333+0000","last_fullsized":"2024-09-11T14:42:13.084333+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"27'294","reported_seq":330,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:14.603190+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:42:14.603190+0000","last_peered":"2024-09-11T14:42:14.603190+0000","last_clean":"2024-09-11T14:42:14.603190+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:42:14.603190+0000","last_undegraded":"2024-09-11T14:42:14.603190+0000","last_fullsized":"2024-09-11T14:42:14.603190+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":294,"log_dups_size":0,"ondisk_log_size":294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19267584,"num_objects":294,"num_object_clones":0,"num_object_copies":882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":294,"num_write_kb":18816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.179633+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:42:13.179633+0000","last_peered":"2024-09-11T14:42:13.179633+0000","last_clean":"2024-09-11T14:42:13.179633+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:42:13.179633+0000","last_undegraded":"2024-09-11T14:42:13.179633+0000","last_fullsized":"2024-09-11T14:42:13.179633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"27'329","reported_seq":365,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:14.364010+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:42:14.364010+0000","last_peered":"2024-09-11T14:42:14.364010+0000","last_clean":"2024-09-11T14:42:14.364010+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:42:14.364010+0000","last_undegraded":"2024-09-11T14:42:14.364010+0000","last_fullsized":"2024-09-11T14:42:14.364010+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21561344,"num_objects":329,"num_object_clones":0,"num_object_copies":987,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":329,"num_write_kb":21056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":51,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.082423+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:42:13.082423+0000","last_peered":"2024-09-11T14:42:13.082423+0000","last_clean":"2024-09-11T14:42:13.082423+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:42:13.082423+0000","last_undegraded":"2024-09-11T14:42:13.082423+0000","last_fullsized":"2024-09-11T14:42:13.082423+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"27'316","reported_seq":352,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:15.696272+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:42:15.696272+0000","last_peered":"2024-09-11T14:42:15.696272+0000","last_clean":"2024-09-11T14:42:15.696272+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:42:15.696272+0000","last_undegraded":"2024-09-11T14:42:15.696272+0000","last_fullsized":"2024-09-11T14:42:15.696272+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20709376,"num_objects":316,"num_object_clones":0,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":316,"num_write_kb":20224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"27'303","reported_seq":339,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:14.208182+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:42:14.208182+0000","last_peered":"2024-09-11T14:42:14.208182+0000","last_clean":"2024-09-11T14:42:14.208182+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:42:14.208182+0000","last_undegraded":"2024-09-11T14:42:14.208182+0000","last_fullsized":"2024-09-11T14:42:14.208182+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19857408,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":303,"num_write_kb":19392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111507+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:42:13.111507+0000","last_peered":"2024-09-11T14:42:13.111507+0000","last_clean":"2024-09-11T14:42:13.111507+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:42:13.111507+0000","last_undegraded":"2024-09-11T14:42:13.111507+0000","last_fullsized":"2024-09-11T14:42:13.111507+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"27'321","reported_seq":357,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:15.639635+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:42:15.639635+0000","last_peered":"2024-09-11T14:42:15.639635+0000","last_clean":"2024-09-11T14:42:15.639635+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:42:15.639635+0000","last_undegraded":"2024-09-11T14:42:15.639635+0000","last_fullsized":"2024-09-11T14:42:15.639635+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21037056,"num_objects":321,"num_object_clones":0,"num_object_copies":963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":321,"num_write_kb":20544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.086961+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:42:13.086961+0000","last_peered":"2024-09-11T14:42:13.086961+0000","last_clean":"2024-09-11T14:42:13.086961+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:42:13.086961+0000","last_undegraded":"2024-09-11T14:42:13.086961+0000","last_fullsized":"2024-09-11T14:42:13.086961+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"27'334","reported_seq":370,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:14.543597+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:42:14.543597+0000","last_peered":"2024-09-11T14:42:14.543597+0000","last_clean":"2024-09-11T14:42:14.543597+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:42:14.543597+0000","last_undegraded":"2024-09-11T14:42:14.543597+0000","last_fullsized":"2024-09-11T14:42:14.543597+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21889024,"num_objects":334,"num_object_clones":0,"num_object_copies":1002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":334,"num_write_kb":21376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111171+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:42:13.111171+0000","last_peered":"2024-09-11T14:42:13.111171+0000","last_clean":"2024-09-11T14:42:13.111171+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:42:13.111171+0000","last_undegraded":"2024-09-11T14:42:13.111171+0000","last_fullsized":"2024-09-11T14:42:13.111171+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"27'331","reported_seq":367,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:15.579009+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:42:15.579009+0000","last_peered":"2024-09-11T14:42:15.579009+0000","last_clean":"2024-09-11T14:42:15.579009+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:42:15.579009+0000","last_undegraded":"2024-09-11T14:42:15.579009+0000","last_fullsized":"2024-09-11T14:42:15.579009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21692416,"num_objects":331,"num_object_clones":0,"num_object_copies":993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":331,"num_write_kb":21184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.181385+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:42:13.181385+0000","last_peered":"2024-09-11T14:42:13.181385+0000","last_clean":"2024-09-11T14:42:13.181385+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:42:13.181385+0000","last_undegraded":"2024-09-11T14:42:13.181385+0000","last_fullsized":"2024-09-11T14:42:13.181385+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":336855040,"num_objects":5140,"num_object_clones":0,"num_object_copies":15420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5140,"num_write_kb":328960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":212169330688,"internally_reserved":0,"allocated":2042163200,"data_stored":2042163200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5140,"ondisk_log_size":5140,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":212169723904,"internally_reserved":0,"allocated":2041769984,"data_stored":2041769984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":52965675008,"internally_reserved":0,"allocated":587198464,"data_stored":587198464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673029,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":463012,"kb_used_data":463012,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51834716,"statfs":{"total":53552873472,"available":53078749184,"internally_reserved":0,"allocated":474124288,"data_stored":474124288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705733,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":564292,"kb_used_data":564292,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51733436,"statfs":{"total":53552873472,"available":52975038464,"internally_reserved":0,"allocated":577835008,"data_stored":577835008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705733,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":487636,"kb_used_data":487636,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51810092,"statfs":{"total":53552873472,"available":53053534208,"internally_reserved":0,"allocated":499339264,"data_stored":499339264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705733,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":445592,"kb_used_data":445592,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51852136,"statfs":{"total":53552873472,"available":53096587264,"internally_reserved":0,"allocated":456286208,"data_stored":456286208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":52965675008,"internally_reserved":0,"allocated":587198464,"data_stored":587198464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53089054720,"internally_reserved":0,"allocated":463818752,"data_stored":463818752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53044854784,"internally_reserved":0,"allocated":508018688,"data_stored":508018688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53070139392,"internally_reserved":0,"allocated":482734080,"data_stored":482734080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":52965675008,"internally_reserved":0,"allocated":587198464,"data_stored":587198464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":53088923648,"internally_reserved":0,"allocated":463949824,"data_stored":463949824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":53044723712,"internally_reserved":0,"allocated":508149760,"data_stored":508149760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":53070139392,"internally_reserved":0,"allocated":482734080,"data_stored":482734080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":52965543936,"internally_reserved":0,"allocated":587329536,"data_stored":587329536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:42:20.697 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:42:20.857 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:20.854+0000 7f8bf0791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:20.863 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:20.862+0000 7f8bf0791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:20.909 INFO:tasks.radosbench.radosbench.0.smithi181.stdout:2024-09-11T14:42:20.909064+0000 min lat: 0.0249053 max lat: 1.01958 avg lat: 0.168041 2024-09-11T14:42:20.909 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T14:42:20.910 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 60 16 5721 5705 5.94211 5.75 0.313642 0.168041 2024-09-11T14:42:21.909 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 61 16 5813 5797 5.93895 5.75 0.129702 0.168123 2024-09-11T14:42:22.911 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 62 16 5916 5900 5.94698 6.4375 0.146104 0.167932 2024-09-11T14:42:23.411 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:42:23.582 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:23.578+0000 7fe312391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:23.588 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:23.587+0000 7fe312391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:23.909 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 63 16 6021 6005 5.95674 6.5625 0.131432 0.167619 2024-09-11T14:42:24.912 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 64 16 6117 6101 5.95741 6 0.107997 0.167577 2024-09-11T14:42:25.618 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:42:25.618 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:42:25.618 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:42:25.749 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:25.742+0000 7faecf646640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:25.754 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:25.752+0000 7faecf646640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:25.910 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 65 16 6200 6184 5.94555 5.1875 0.367441 0.167864 2024-09-11T14:42:25.954 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:42:25.955 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:42:25.971 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2024-09-11T14:42:24.721287+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":370999859,"num_objects":5656,"num_object_clones":0,"num_object_copies":16968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":5733,"num_write_kb":362836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5707,"ondisk_log_size":5707,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":2151996,"kb_used_data":2151996,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":207038916,"statfs":{"total":214211493888,"available":212007849984,"internally_reserved":0,"allocated":2203643904,"data_stored":2203643904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":64552960,"num_objects":985,"num_object_clones":0,"num_object_copies":2955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":985,"num_write_kb":63040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.781155"},"pg_stats":[{"pgid":"3.f","version":"27'326","reported_seq":362,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:19.666061+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:42:19.666061+0000","last_peered":"2024-09-11T14:42:19.666061+0000","last_clean":"2024-09-11T14:42:19.666061+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:42:19.666061+0000","last_undegraded":"2024-09-11T14:42:19.666061+0000","last_fullsized":"2024-09-11T14:42:19.666061+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21364736,"num_objects":326,"num_object_clones":0,"num_object_copies":978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":326,"num_write_kb":20864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"27'398","reported_seq":434,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:20.648399+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:42:20.648399+0000","last_peered":"2024-09-11T14:42:20.648399+0000","last_clean":"2024-09-11T14:42:20.648399+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:42:20.648399+0000","last_undegraded":"2024-09-11T14:42:20.648399+0000","last_fullsized":"2024-09-11T14:42:20.648399+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26083328,"num_objects":398,"num_object_clones":0,"num_object_copies":1194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":398,"num_write_kb":25472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"27'337","reported_seq":373,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:19.647549+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:42:19.647549+0000","last_peered":"2024-09-11T14:42:19.647549+0000","last_clean":"2024-09-11T14:42:19.647549+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:42:19.647549+0000","last_undegraded":"2024-09-11T14:42:19.647549+0000","last_fullsized":"2024-09-11T14:42:19.647549+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22085632,"num_objects":337,"num_object_clones":0,"num_object_copies":1011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":337,"num_write_kb":21568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"27'369","reported_seq":405,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:19.801056+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:42:19.801056+0000","last_peered":"2024-09-11T14:42:19.801056+0000","last_clean":"2024-09-11T14:42:19.801056+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:42:19.801056+0000","last_undegraded":"2024-09-11T14:42:19.801056+0000","last_fullsized":"2024-09-11T14:42:19.801056+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24182784,"num_objects":369,"num_object_clones":0,"num_object_copies":1107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":369,"num_write_kb":23616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"27'356","reported_seq":392,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:19.506111+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:42:19.506111+0000","last_peered":"2024-09-11T14:42:19.506111+0000","last_clean":"2024-09-11T14:42:19.506111+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:42:19.506111+0000","last_undegraded":"2024-09-11T14:42:19.506111+0000","last_fullsized":"2024-09-11T14:42:19.506111+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23330816,"num_objects":356,"num_object_clones":0,"num_object_copies":1068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":356,"num_write_kb":22784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"27'391","reported_seq":427,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:19.589488+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:42:19.589488+0000","last_peered":"2024-09-11T14:42:19.589488+0000","last_clean":"2024-09-11T14:42:19.589488+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:42:19.589488+0000","last_undegraded":"2024-09-11T14:42:19.589488+0000","last_fullsized":"2024-09-11T14:42:19.589488+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":391,"log_dups_size":0,"ondisk_log_size":391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624576,"num_objects":391,"num_object_clones":0,"num_object_copies":1173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":391,"num_write_kb":25024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"27'355","reported_seq":391,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:19.328466+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:42:19.328466+0000","last_peered":"2024-09-11T14:42:19.328466+0000","last_clean":"2024-09-11T14:42:19.328466+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:42:19.328466+0000","last_undegraded":"2024-09-11T14:42:19.328466+0000","last_fullsized":"2024-09-11T14:42:19.328466+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23265280,"num_objects":355,"num_object_clones":0,"num_object_copies":1065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":355,"num_write_kb":22720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"27'314","reported_seq":350,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:19.450159+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:42:19.450159+0000","last_peered":"2024-09-11T14:42:19.450159+0000","last_clean":"2024-09-11T14:42:19.450159+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:42:19.450159+0000","last_undegraded":"2024-09-11T14:42:19.450159+0000","last_fullsized":"2024-09-11T14:42:19.450159+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20578304,"num_objects":314,"num_object_clones":0,"num_object_copies":942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":314,"num_write_kb":20096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":121,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084543+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:42:13.084543+0000","last_peered":"2024-09-11T14:42:13.084543+0000","last_clean":"2024-09-11T14:42:13.084543+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:42:13.084543+0000","last_undegraded":"2024-09-11T14:42:13.084543+0000","last_fullsized":"2024-09-11T14:42:13.084543+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":50,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.184283+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:42:13.184283+0000","last_peered":"2024-09-11T14:42:13.184283+0000","last_clean":"2024-09-11T14:42:13.184283+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:42:13.184283+0000","last_undegraded":"2024-09-11T14:42:13.184283+0000","last_fullsized":"2024-09-11T14:42:13.184283+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"27'358","reported_seq":394,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:19.777730+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:42:19.777730+0000","last_peered":"2024-09-11T14:42:19.777730+0000","last_clean":"2024-09-11T14:42:19.777730+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:42:19.777730+0000","last_undegraded":"2024-09-11T14:42:19.777730+0000","last_fullsized":"2024-09-11T14:42:19.777730+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23461888,"num_objects":358,"num_object_clones":0,"num_object_copies":1074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":358,"num_write_kb":22912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084333+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:42:13.084333+0000","last_peered":"2024-09-11T14:42:13.084333+0000","last_clean":"2024-09-11T14:42:13.084333+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:42:13.084333+0000","last_undegraded":"2024-09-11T14:42:13.084333+0000","last_fullsized":"2024-09-11T14:42:13.084333+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"27'329","reported_seq":365,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:19.499018+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:42:19.499018+0000","last_peered":"2024-09-11T14:42:19.499018+0000","last_clean":"2024-09-11T14:42:19.499018+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:42:19.499018+0000","last_undegraded":"2024-09-11T14:42:19.499018+0000","last_fullsized":"2024-09-11T14:42:19.499018+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21561344,"num_objects":329,"num_object_clones":0,"num_object_copies":987,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":329,"num_write_kb":21056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.179633+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:42:13.179633+0000","last_peered":"2024-09-11T14:42:13.179633+0000","last_clean":"2024-09-11T14:42:13.179633+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:42:13.179633+0000","last_undegraded":"2024-09-11T14:42:13.179633+0000","last_fullsized":"2024-09-11T14:42:13.179633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"27'358","reported_seq":394,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:19.293177+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:42:19.293177+0000","last_peered":"2024-09-11T14:42:19.293177+0000","last_clean":"2024-09-11T14:42:19.293177+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:42:19.293177+0000","last_undegraded":"2024-09-11T14:42:19.293177+0000","last_fullsized":"2024-09-11T14:42:19.293177+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23461888,"num_objects":358,"num_object_clones":0,"num_object_copies":1074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":358,"num_write_kb":22912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":51,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.082423+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:42:13.082423+0000","last_peered":"2024-09-11T14:42:13.082423+0000","last_clean":"2024-09-11T14:42:13.082423+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:42:13.082423+0000","last_undegraded":"2024-09-11T14:42:13.082423+0000","last_fullsized":"2024-09-11T14:42:13.082423+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"27'347","reported_seq":383,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:20.756556+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:42:20.756556+0000","last_peered":"2024-09-11T14:42:20.756556+0000","last_clean":"2024-09-11T14:42:20.756556+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:42:20.756556+0000","last_undegraded":"2024-09-11T14:42:20.756556+0000","last_fullsized":"2024-09-11T14:42:20.756556+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22740992,"num_objects":347,"num_object_clones":0,"num_object_copies":1041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":347,"num_write_kb":22208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"27'340","reported_seq":376,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:19.536430+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:42:19.536430+0000","last_peered":"2024-09-11T14:42:19.536430+0000","last_clean":"2024-09-11T14:42:19.536430+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:42:19.536430+0000","last_undegraded":"2024-09-11T14:42:19.536430+0000","last_fullsized":"2024-09-11T14:42:19.536430+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":340,"log_dups_size":0,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22282240,"num_objects":340,"num_object_clones":0,"num_object_copies":1020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":340,"num_write_kb":21760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111507+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:42:13.111507+0000","last_peered":"2024-09-11T14:42:13.111507+0000","last_clean":"2024-09-11T14:42:13.111507+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:42:13.111507+0000","last_undegraded":"2024-09-11T14:42:13.111507+0000","last_fullsized":"2024-09-11T14:42:13.111507+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"27'348","reported_seq":384,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:20.703532+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:42:20.703532+0000","last_peered":"2024-09-11T14:42:20.703532+0000","last_clean":"2024-09-11T14:42:20.703532+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:42:20.703532+0000","last_undegraded":"2024-09-11T14:42:20.703532+0000","last_fullsized":"2024-09-11T14:42:20.703532+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22806528,"num_objects":348,"num_object_clones":0,"num_object_copies":1044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":348,"num_write_kb":22272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.086961+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:42:13.086961+0000","last_peered":"2024-09-11T14:42:13.086961+0000","last_clean":"2024-09-11T14:42:13.086961+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:42:13.086961+0000","last_undegraded":"2024-09-11T14:42:13.086961+0000","last_fullsized":"2024-09-11T14:42:13.086961+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"27'360","reported_seq":396,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:19.307021+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:42:19.307021+0000","last_peered":"2024-09-11T14:42:19.307021+0000","last_clean":"2024-09-11T14:42:19.307021+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:42:19.307021+0000","last_undegraded":"2024-09-11T14:42:19.307021+0000","last_fullsized":"2024-09-11T14:42:19.307021+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":360,"log_dups_size":0,"ondisk_log_size":360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23592960,"num_objects":360,"num_object_clones":0,"num_object_copies":1080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":360,"num_write_kb":23040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111171+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:42:13.111171+0000","last_peered":"2024-09-11T14:42:13.111171+0000","last_clean":"2024-09-11T14:42:13.111171+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:42:13.111171+0000","last_undegraded":"2024-09-11T14:42:13.111171+0000","last_fullsized":"2024-09-11T14:42:13.111171+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"27'366","reported_seq":402,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:20.636139+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:42:20.636139+0000","last_peered":"2024-09-11T14:42:20.636139+0000","last_clean":"2024-09-11T14:42:20.636139+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:42:20.636139+0000","last_undegraded":"2024-09-11T14:42:20.636139+0000","last_fullsized":"2024-09-11T14:42:20.636139+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23986176,"num_objects":366,"num_object_clones":0,"num_object_copies":1098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":366,"num_write_kb":23424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.181385+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:42:13.181385+0000","last_peered":"2024-09-11T14:42:13.181385+0000","last_clean":"2024-09-11T14:42:13.181385+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:42:13.181385+0000","last_undegraded":"2024-09-11T14:42:13.181385+0000","last_fullsized":"2024-09-11T14:42:13.181385+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":370409472,"num_objects":5652,"num_object_clones":0,"num_object_copies":16956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5652,"num_write_kb":361728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":211969728512,"internally_reserved":0,"allocated":2241765376,"data_stored":2241765376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5652,"ondisk_log_size":5652,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":211970121728,"internally_reserved":0,"allocated":2241372160,"data_stored":2241372160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":52908171264,"internally_reserved":0,"allocated":644702208,"data_stored":644702208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673034,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":506304,"kb_used_data":506304,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51791424,"statfs":{"total":53552873472,"available":53034418176,"internally_reserved":0,"allocated":518455296,"data_stored":518455296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705738,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":619168,"kb_used_data":619168,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51678560,"statfs":{"total":53552873472,"available":52918845440,"internally_reserved":0,"allocated":634028032,"data_stored":634028032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705738,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":535000,"kb_used_data":535000,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51762728,"statfs":{"total":53552873472,"available":53005033472,"internally_reserved":0,"allocated":547840000,"data_stored":547840000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705738,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":491524,"kb_used_data":491524,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51806204,"statfs":{"total":53552873472,"available":53049552896,"internally_reserved":0,"allocated":503320576,"data_stored":503320576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":52908171264,"internally_reserved":0,"allocated":644702208,"data_stored":644702208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53041143808,"internally_reserved":0,"allocated":511729664,"data_stored":511729664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":52994748416,"internally_reserved":0,"allocated":558125056,"data_stored":558125056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53026193408,"internally_reserved":0,"allocated":526680064,"data_stored":526680064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":52908036096,"internally_reserved":0,"allocated":644837376,"data_stored":644837376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":53041012736,"internally_reserved":0,"allocated":511860736,"data_stored":511860736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":52994748416,"internally_reserved":0,"allocated":558125056,"data_stored":558125056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":53025931264,"internally_reserved":0,"allocated":526942208,"data_stored":526942208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":52908036096,"internally_reserved":0,"allocated":644837376,"data_stored":644837376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:42:26.191 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:42:26.311 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:26.307+0000 7f8baa391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:26.316 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:26.314+0000 7f8baa391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:26.910 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 66 16 6279 6263 5.93027 4.9375 0.0677321 0.168197 2024-09-11T14:42:27.910 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 67 16 6383 6367 5.93877 6.5 0.186011 0.16815 2024-09-11T14:42:28.603 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:42:28.732 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:28.728+0000 7f5c57951640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:28.738 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:28.737+0000 7f5c57951640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:28.910 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 68 16 6472 6456 5.93322 5.5625 0.0781163 0.168324 2024-09-11T14:42:29.910 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 69 16 6564 6548 5.93056 5.75 0.400916 0.168235 2024-09-11T14:42:30.910 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 70 16 6657 6641 5.92887 5.8125 0.105895 0.168279 2024-09-11T14:42:30.973 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:42:30.973 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:42:30.974 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:42:31.128 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:31.121+0000 7f1874791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:31.134 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:31.133+0000 7f1874791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:31.160 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:42:31.328 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:31.322+0000 7f1b3fe4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:31.335 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:31.333+0000 7f1b3fe4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:31.515 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:42:31.516 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:42:31.533 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-09-11T14:42:30.722524+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":423690803,"num_objects":6460,"num_object_clones":0,"num_object_copies":19380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":6537,"num_write_kb":414292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6511,"ondisk_log_size":6511,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":2479132,"kb_used_data":2479132,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":206711780,"statfs":{"total":214211493888,"available":211672862720,"internally_reserved":0,"allocated":2538631168,"data_stored":2538631168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":86245376,"num_objects":1316,"num_object_clones":0,"num_object_copies":3948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1316,"num_write_kb":84224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002526"},"pg_stats":[{"pgid":"3.f","version":"27'385","reported_seq":421,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:29.433975+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:42:29.433975+0000","last_peered":"2024-09-11T14:42:29.433975+0000","last_clean":"2024-09-11T14:42:29.433975+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:42:29.433975+0000","last_undegraded":"2024-09-11T14:42:29.433975+0000","last_fullsized":"2024-09-11T14:42:29.433975+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":385,"log_dups_size":0,"ondisk_log_size":385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25231360,"num_objects":385,"num_object_clones":0,"num_object_copies":1155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":385,"num_write_kb":24640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"27'433","reported_seq":469,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:25.745854+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:42:25.745854+0000","last_peered":"2024-09-11T14:42:25.745854+0000","last_clean":"2024-09-11T14:42:25.745854+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:42:25.745854+0000","last_undegraded":"2024-09-11T14:42:25.745854+0000","last_fullsized":"2024-09-11T14:42:25.745854+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":433,"log_dups_size":0,"ondisk_log_size":433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28377088,"num_objects":433,"num_object_clones":0,"num_object_copies":1299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":433,"num_write_kb":27712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"27'398","reported_seq":434,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:29.659843+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:42:29.659843+0000","last_peered":"2024-09-11T14:42:29.659843+0000","last_clean":"2024-09-11T14:42:29.659843+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:42:29.659843+0000","last_undegraded":"2024-09-11T14:42:29.659843+0000","last_fullsized":"2024-09-11T14:42:29.659843+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26083328,"num_objects":398,"num_object_clones":0,"num_object_copies":1194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":398,"num_write_kb":25472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"27'417","reported_seq":453,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:29.189305+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:42:29.189305+0000","last_peered":"2024-09-11T14:42:29.189305+0000","last_clean":"2024-09-11T14:42:29.189305+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:42:29.189305+0000","last_undegraded":"2024-09-11T14:42:29.189305+0000","last_fullsized":"2024-09-11T14:42:29.189305+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":417,"log_dups_size":0,"ondisk_log_size":417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27328512,"num_objects":417,"num_object_clones":0,"num_object_copies":1251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":417,"num_write_kb":26688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"27'414","reported_seq":450,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:29.755415+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:42:29.755415+0000","last_peered":"2024-09-11T14:42:29.755415+0000","last_clean":"2024-09-11T14:42:29.755415+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:42:29.755415+0000","last_undegraded":"2024-09-11T14:42:29.755415+0000","last_fullsized":"2024-09-11T14:42:29.755415+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":414,"log_dups_size":0,"ondisk_log_size":414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27131904,"num_objects":414,"num_object_clones":0,"num_object_copies":1242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":414,"num_write_kb":26496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"27'444","reported_seq":480,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:29.541274+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:42:29.541274+0000","last_peered":"2024-09-11T14:42:29.541274+0000","last_clean":"2024-09-11T14:42:29.541274+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:42:29.541274+0000","last_undegraded":"2024-09-11T14:42:29.541274+0000","last_fullsized":"2024-09-11T14:42:29.541274+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29097984,"num_objects":444,"num_object_clones":0,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":444,"num_write_kb":28416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"27'423","reported_seq":459,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:29.628582+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:42:29.628582+0000","last_peered":"2024-09-11T14:42:29.628582+0000","last_clean":"2024-09-11T14:42:29.628582+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:42:29.628582+0000","last_undegraded":"2024-09-11T14:42:29.628582+0000","last_fullsized":"2024-09-11T14:42:29.628582+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":423,"log_dups_size":0,"ondisk_log_size":423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27721728,"num_objects":423,"num_object_clones":0,"num_object_copies":1269,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":423,"num_write_kb":27072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"27'384","reported_seq":420,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:29.334093+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:42:29.334093+0000","last_peered":"2024-09-11T14:42:29.334093+0000","last_clean":"2024-09-11T14:42:29.334093+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:42:29.334093+0000","last_undegraded":"2024-09-11T14:42:29.334093+0000","last_fullsized":"2024-09-11T14:42:29.334093+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":384,"num_object_clones":0,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":384,"num_write_kb":24576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":121,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084543+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:42:13.084543+0000","last_peered":"2024-09-11T14:42:13.084543+0000","last_clean":"2024-09-11T14:42:13.084543+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:42:13.084543+0000","last_undegraded":"2024-09-11T14:42:13.084543+0000","last_fullsized":"2024-09-11T14:42:13.084543+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":50,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.184283+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:42:13.184283+0000","last_peered":"2024-09-11T14:42:13.184283+0000","last_clean":"2024-09-11T14:42:13.184283+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:42:13.184283+0000","last_undegraded":"2024-09-11T14:42:13.184283+0000","last_fullsized":"2024-09-11T14:42:13.184283+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"27'407","reported_seq":443,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:29.571811+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:42:29.571811+0000","last_peered":"2024-09-11T14:42:29.571811+0000","last_clean":"2024-09-11T14:42:29.571811+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:42:29.571811+0000","last_undegraded":"2024-09-11T14:42:29.571811+0000","last_fullsized":"2024-09-11T14:42:29.571811+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26673152,"num_objects":407,"num_object_clones":0,"num_object_copies":1221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":407,"num_write_kb":26048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084333+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:42:13.084333+0000","last_peered":"2024-09-11T14:42:13.084333+0000","last_clean":"2024-09-11T14:42:13.084333+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:42:13.084333+0000","last_undegraded":"2024-09-11T14:42:13.084333+0000","last_fullsized":"2024-09-11T14:42:13.084333+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"27'389","reported_seq":425,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:29.393516+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:42:29.393516+0000","last_peered":"2024-09-11T14:42:29.393516+0000","last_clean":"2024-09-11T14:42:29.393516+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:42:29.393516+0000","last_undegraded":"2024-09-11T14:42:29.393516+0000","last_fullsized":"2024-09-11T14:42:29.393516+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25493504,"num_objects":389,"num_object_clones":0,"num_object_copies":1167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":389,"num_write_kb":24896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.179633+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:42:13.179633+0000","last_peered":"2024-09-11T14:42:13.179633+0000","last_clean":"2024-09-11T14:42:13.179633+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:42:13.179633+0000","last_undegraded":"2024-09-11T14:42:13.179633+0000","last_fullsized":"2024-09-11T14:42:13.179633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"27'404","reported_seq":440,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:29.011798+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:42:29.011798+0000","last_peered":"2024-09-11T14:42:29.011798+0000","last_clean":"2024-09-11T14:42:29.011798+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:42:29.011798+0000","last_undegraded":"2024-09-11T14:42:29.011798+0000","last_fullsized":"2024-09-11T14:42:29.011798+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":404,"log_dups_size":0,"ondisk_log_size":404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26476544,"num_objects":404,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":404,"num_write_kb":25856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":51,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.082423+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:42:13.082423+0000","last_peered":"2024-09-11T14:42:13.082423+0000","last_clean":"2024-09-11T14:42:13.082423+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:42:13.082423+0000","last_undegraded":"2024-09-11T14:42:13.082423+0000","last_fullsized":"2024-09-11T14:42:13.082423+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"27'367","reported_seq":403,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:24.668884+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:42:24.668884+0000","last_peered":"2024-09-11T14:42:24.668884+0000","last_clean":"2024-09-11T14:42:24.668884+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:42:24.668884+0000","last_undegraded":"2024-09-11T14:42:24.668884+0000","last_fullsized":"2024-09-11T14:42:24.668884+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24051712,"num_objects":367,"num_object_clones":0,"num_object_copies":1101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":367,"num_write_kb":23488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"27'403","reported_seq":439,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:29.752896+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:42:29.752896+0000","last_peered":"2024-09-11T14:42:29.752896+0000","last_clean":"2024-09-11T14:42:29.752896+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:42:29.752896+0000","last_undegraded":"2024-09-11T14:42:29.752896+0000","last_fullsized":"2024-09-11T14:42:29.752896+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26411008,"num_objects":403,"num_object_clones":0,"num_object_copies":1209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":403,"num_write_kb":25792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111507+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:42:13.111507+0000","last_peered":"2024-09-11T14:42:13.111507+0000","last_clean":"2024-09-11T14:42:13.111507+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:42:13.111507+0000","last_undegraded":"2024-09-11T14:42:13.111507+0000","last_fullsized":"2024-09-11T14:42:13.111507+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"27'374","reported_seq":410,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:25.299513+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:42:25.299513+0000","last_peered":"2024-09-11T14:42:25.299513+0000","last_clean":"2024-09-11T14:42:25.299513+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:42:25.299513+0000","last_undegraded":"2024-09-11T14:42:25.299513+0000","last_fullsized":"2024-09-11T14:42:25.299513+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24510464,"num_objects":374,"num_object_clones":0,"num_object_copies":1122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":374,"num_write_kb":23936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.086961+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:42:13.086961+0000","last_peered":"2024-09-11T14:42:13.086961+0000","last_clean":"2024-09-11T14:42:13.086961+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:42:13.086961+0000","last_undegraded":"2024-09-11T14:42:13.086961+0000","last_fullsized":"2024-09-11T14:42:13.086961+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"27'416","reported_seq":452,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:29.601520+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:42:29.601520+0000","last_peered":"2024-09-11T14:42:29.601520+0000","last_clean":"2024-09-11T14:42:29.601520+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:42:29.601520+0000","last_undegraded":"2024-09-11T14:42:29.601520+0000","last_fullsized":"2024-09-11T14:42:29.601520+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27262976,"num_objects":416,"num_object_clones":0,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":416,"num_write_kb":26624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111171+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:42:13.111171+0000","last_peered":"2024-09-11T14:42:13.111171+0000","last_clean":"2024-09-11T14:42:13.111171+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:42:13.111171+0000","last_undegraded":"2024-09-11T14:42:13.111171+0000","last_fullsized":"2024-09-11T14:42:13.111171+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"27'398","reported_seq":434,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:25.671212+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:42:25.671212+0000","last_peered":"2024-09-11T14:42:25.671212+0000","last_clean":"2024-09-11T14:42:25.671212+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:42:25.671212+0000","last_undegraded":"2024-09-11T14:42:25.671212+0000","last_fullsized":"2024-09-11T14:42:25.671212+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26083328,"num_objects":398,"num_object_clones":0,"num_object_copies":1194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":398,"num_write_kb":25472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.181385+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:42:13.181385+0000","last_peered":"2024-09-11T14:42:13.181385+0000","last_clean":"2024-09-11T14:42:13.181385+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:42:13.181385+0000","last_undegraded":"2024-09-11T14:42:13.181385+0000","last_fullsized":"2024-09-11T14:42:13.181385+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":423100416,"num_objects":6456,"num_object_clones":0,"num_object_copies":19368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6456,"num_write_kb":413184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":211637780480,"internally_reserved":0,"allocated":2573713408,"data_stored":2573713408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6456,"ondisk_log_size":6456,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":211637780480,"internally_reserved":0,"allocated":2573713408,"data_stored":2573713408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":52799160320,"internally_reserved":0,"allocated":753713152,"data_stored":753713152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673039,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":551764,"kb_used_data":551764,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51745964,"statfs":{"total":53552873472,"available":52987867136,"internally_reserved":0,"allocated":565006336,"data_stored":565006336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705748,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":725492,"kb_used_data":725492,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51572236,"statfs":{"total":53552873472,"available":52809969664,"internally_reserved":0,"allocated":742903808,"data_stored":742903808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705748,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":629196,"kb_used_data":629196,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51668532,"statfs":{"total":53552873472,"available":52908576768,"internally_reserved":0,"allocated":644296704,"data_stored":644296704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705748,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":572680,"kb_used_data":572680,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51725048,"statfs":{"total":53552873472,"available":52966449152,"internally_reserved":0,"allocated":586424320,"data_stored":586424320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":52799160320,"internally_reserved":0,"allocated":753713152,"data_stored":753713152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":52958171136,"internally_reserved":0,"allocated":594702336,"data_stored":594702336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":52899348480,"internally_reserved":0,"allocated":653524992,"data_stored":653524992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":52981100544,"internally_reserved":0,"allocated":571772928,"data_stored":571772928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":52799160320,"internally_reserved":0,"allocated":753713152,"data_stored":753713152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":52958171136,"internally_reserved":0,"allocated":594702336,"data_stored":594702336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":52899348480,"internally_reserved":0,"allocated":653524992,"data_stored":653524992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":52981100544,"internally_reserved":0,"allocated":571772928,"data_stored":571772928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":52799160320,"internally_reserved":0,"allocated":753713152,"data_stored":753713152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:42:31.910 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 71 16 6751 6735 5.9281 5.875 0.225265 0.168396 2024-09-11T14:42:32.910 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 72 16 6846 6830 5.92822 5.9375 0.0807357 0.168414 2024-09-11T14:42:33.756 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:42:33.905 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:33.901+0000 7fb5862c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:33.910 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 73 16 6940 6924 5.92749 5.875 0.0728906 0.168432 2024-09-11T14:42:33.910 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:33.909+0000 7fb5862c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:34.910 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 74 16 7023 7007 5.91748 5.1875 0.423252 0.168633 2024-09-11T14:42:35.911 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 75 16 7104 7088 5.90607 5.0625 0.138313 0.169042 2024-09-11T14:42:36.338 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:42:36.500 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:36.496+0000 7f62b8d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:36.506 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:36.505+0000 7f62b8d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:36.536 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:42:36.536 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:42:36.536 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:42:36.671 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:36.664+0000 7f2237111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:36.675 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:36.674+0000 7f2237111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:36.911 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 76 16 7193 7177 5.90155 5.5625 0.104095 0.169139 2024-09-11T14:42:37.008 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:42:37.009 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:42:37.035 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2024-09-11T14:42:36.723839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":461898291,"num_objects":7043,"num_object_clones":0,"num_object_copies":21129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":7120,"num_write_kb":451604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7094,"ondisk_log_size":7094,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":2699368,"kb_used_data":2699368,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":206491544,"statfs":{"total":214211493888,"available":211447341056,"internally_reserved":0,"allocated":2764152832,"data_stored":2764152832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":90898432,"num_objects":1387,"num_object_clones":0,"num_object_copies":4161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1387,"num_write_kb":88768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002552"},"pg_stats":[{"pgid":"3.f","version":"27'416","reported_seq":452,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:34.499741+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:42:34.499741+0000","last_peered":"2024-09-11T14:42:34.499741+0000","last_clean":"2024-09-11T14:42:34.499741+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:42:34.499741+0000","last_undegraded":"2024-09-11T14:42:34.499741+0000","last_fullsized":"2024-09-11T14:42:34.499741+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27262976,"num_objects":416,"num_object_clones":0,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":416,"num_write_kb":26624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"27'495","reported_seq":531,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:35.445372+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:42:35.445372+0000","last_peered":"2024-09-11T14:42:35.445372+0000","last_clean":"2024-09-11T14:42:35.445372+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:42:35.445372+0000","last_undegraded":"2024-09-11T14:42:35.445372+0000","last_fullsized":"2024-09-11T14:42:35.445372+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":495,"log_dups_size":0,"ondisk_log_size":495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32440320,"num_objects":495,"num_object_clones":0,"num_object_copies":1485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":495,"num_write_kb":31680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"27'428","reported_seq":464,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:34.713216+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:42:34.713216+0000","last_peered":"2024-09-11T14:42:34.713216+0000","last_clean":"2024-09-11T14:42:34.713216+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:42:34.713216+0000","last_undegraded":"2024-09-11T14:42:34.713216+0000","last_fullsized":"2024-09-11T14:42:34.713216+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":428,"log_dups_size":0,"ondisk_log_size":428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28049408,"num_objects":428,"num_object_clones":0,"num_object_copies":1284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":428,"num_write_kb":27392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"27'436","reported_seq":472,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:34.354794+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:42:34.354794+0000","last_peered":"2024-09-11T14:42:34.354794+0000","last_clean":"2024-09-11T14:42:34.354794+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:42:34.354794+0000","last_undegraded":"2024-09-11T14:42:34.354794+0000","last_fullsized":"2024-09-11T14:42:34.354794+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":436,"log_dups_size":0,"ondisk_log_size":436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28573696,"num_objects":436,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":436,"num_write_kb":27904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"27'434","reported_seq":470,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:34.102989+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:42:34.102989+0000","last_peered":"2024-09-11T14:42:34.102989+0000","last_clean":"2024-09-11T14:42:34.102989+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:42:34.102989+0000","last_undegraded":"2024-09-11T14:42:34.102989+0000","last_fullsized":"2024-09-11T14:42:34.102989+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":434,"log_dups_size":0,"ondisk_log_size":434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28442624,"num_objects":434,"num_object_clones":0,"num_object_copies":1302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":434,"num_write_kb":27776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"27'478","reported_seq":514,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:34.163865+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:42:34.163865+0000","last_peered":"2024-09-11T14:42:34.163865+0000","last_clean":"2024-09-11T14:42:34.163865+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:42:34.163865+0000","last_undegraded":"2024-09-11T14:42:34.163865+0000","last_fullsized":"2024-09-11T14:42:34.163865+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":478,"log_dups_size":0,"ondisk_log_size":478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31326208,"num_objects":478,"num_object_clones":0,"num_object_copies":1434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":478,"num_write_kb":30592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"27'457","reported_seq":493,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:34.727245+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:42:34.727245+0000","last_peered":"2024-09-11T14:42:34.727245+0000","last_clean":"2024-09-11T14:42:34.727245+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:42:34.727245+0000","last_undegraded":"2024-09-11T14:42:34.727245+0000","last_fullsized":"2024-09-11T14:42:34.727245+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":457,"log_dups_size":0,"ondisk_log_size":457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29949952,"num_objects":457,"num_object_clones":0,"num_object_copies":1371,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":457,"num_write_kb":29248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"27'414","reported_seq":450,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:34.542844+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:42:34.542844+0000","last_peered":"2024-09-11T14:42:34.542844+0000","last_clean":"2024-09-11T14:42:34.542844+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:42:34.542844+0000","last_undegraded":"2024-09-11T14:42:34.542844+0000","last_fullsized":"2024-09-11T14:42:34.542844+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":414,"log_dups_size":0,"ondisk_log_size":414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27131904,"num_objects":414,"num_object_clones":0,"num_object_copies":1242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":414,"num_write_kb":26496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":121,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084543+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:42:13.084543+0000","last_peered":"2024-09-11T14:42:13.084543+0000","last_clean":"2024-09-11T14:42:13.084543+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:42:13.084543+0000","last_undegraded":"2024-09-11T14:42:13.084543+0000","last_fullsized":"2024-09-11T14:42:13.084543+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":50,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.184283+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:42:13.184283+0000","last_peered":"2024-09-11T14:42:13.184283+0000","last_clean":"2024-09-11T14:42:13.184283+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:42:13.184283+0000","last_undegraded":"2024-09-11T14:42:13.184283+0000","last_fullsized":"2024-09-11T14:42:13.184283+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"27'433","reported_seq":469,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:34.554735+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:42:34.554735+0000","last_peered":"2024-09-11T14:42:34.554735+0000","last_clean":"2024-09-11T14:42:34.554735+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:42:34.554735+0000","last_undegraded":"2024-09-11T14:42:34.554735+0000","last_fullsized":"2024-09-11T14:42:34.554735+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":433,"log_dups_size":0,"ondisk_log_size":433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28377088,"num_objects":433,"num_object_clones":0,"num_object_copies":1299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":433,"num_write_kb":27712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084333+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:42:13.084333+0000","last_peered":"2024-09-11T14:42:13.084333+0000","last_clean":"2024-09-11T14:42:13.084333+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:42:13.084333+0000","last_undegraded":"2024-09-11T14:42:13.084333+0000","last_fullsized":"2024-09-11T14:42:13.084333+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"27'413","reported_seq":449,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:34.512353+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:42:34.512353+0000","last_peered":"2024-09-11T14:42:34.512353+0000","last_clean":"2024-09-11T14:42:34.512353+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:42:34.512353+0000","last_undegraded":"2024-09-11T14:42:34.512353+0000","last_fullsized":"2024-09-11T14:42:34.512353+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":413,"log_dups_size":0,"ondisk_log_size":413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27066368,"num_objects":413,"num_object_clones":0,"num_object_copies":1239,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":413,"num_write_kb":26432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.179633+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:42:13.179633+0000","last_peered":"2024-09-11T14:42:13.179633+0000","last_clean":"2024-09-11T14:42:13.179633+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:42:13.179633+0000","last_undegraded":"2024-09-11T14:42:13.179633+0000","last_fullsized":"2024-09-11T14:42:13.179633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"27'426","reported_seq":462,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:34.714389+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:42:34.714389+0000","last_peered":"2024-09-11T14:42:34.714389+0000","last_clean":"2024-09-11T14:42:34.714389+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:42:34.714389+0000","last_undegraded":"2024-09-11T14:42:34.714389+0000","last_fullsized":"2024-09-11T14:42:34.714389+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27918336,"num_objects":426,"num_object_clones":0,"num_object_copies":1278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":426,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":51,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.082423+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:42:13.082423+0000","last_peered":"2024-09-11T14:42:13.082423+0000","last_clean":"2024-09-11T14:42:13.082423+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:42:13.082423+0000","last_undegraded":"2024-09-11T14:42:13.082423+0000","last_fullsized":"2024-09-11T14:42:13.082423+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"27'429","reported_seq":465,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:35.673482+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:42:35.673482+0000","last_peered":"2024-09-11T14:42:35.673482+0000","last_clean":"2024-09-11T14:42:35.673482+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:42:35.673482+0000","last_undegraded":"2024-09-11T14:42:35.673482+0000","last_fullsized":"2024-09-11T14:42:35.673482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":429,"log_dups_size":0,"ondisk_log_size":429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28114944,"num_objects":429,"num_object_clones":0,"num_object_copies":1287,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":429,"num_write_kb":27456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"27'442","reported_seq":478,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:34.680335+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:42:34.680335+0000","last_peered":"2024-09-11T14:42:34.680335+0000","last_clean":"2024-09-11T14:42:34.680335+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:42:34.680335+0000","last_undegraded":"2024-09-11T14:42:34.680335+0000","last_fullsized":"2024-09-11T14:42:34.680335+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":442,"log_dups_size":0,"ondisk_log_size":442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28966912,"num_objects":442,"num_object_clones":0,"num_object_copies":1326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":442,"num_write_kb":28288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111507+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:42:13.111507+0000","last_peered":"2024-09-11T14:42:13.111507+0000","last_clean":"2024-09-11T14:42:13.111507+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:42:13.111507+0000","last_undegraded":"2024-09-11T14:42:13.111507+0000","last_fullsized":"2024-09-11T14:42:13.111507+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"27'429","reported_seq":465,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:35.635112+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:42:35.635112+0000","last_peered":"2024-09-11T14:42:35.635112+0000","last_clean":"2024-09-11T14:42:35.635112+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:42:35.635112+0000","last_undegraded":"2024-09-11T14:42:35.635112+0000","last_fullsized":"2024-09-11T14:42:35.635112+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":429,"log_dups_size":0,"ondisk_log_size":429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28114944,"num_objects":429,"num_object_clones":0,"num_object_copies":1287,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":429,"num_write_kb":27456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.086961+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:42:13.086961+0000","last_peered":"2024-09-11T14:42:13.086961+0000","last_clean":"2024-09-11T14:42:13.086961+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:42:13.086961+0000","last_undegraded":"2024-09-11T14:42:13.086961+0000","last_fullsized":"2024-09-11T14:42:13.086961+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"27'448","reported_seq":484,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:34.317177+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:42:34.317177+0000","last_peered":"2024-09-11T14:42:34.317177+0000","last_clean":"2024-09-11T14:42:34.317177+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:42:34.317177+0000","last_undegraded":"2024-09-11T14:42:34.317177+0000","last_fullsized":"2024-09-11T14:42:34.317177+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":448,"log_dups_size":0,"ondisk_log_size":448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":448,"num_object_clones":0,"num_object_copies":1344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":448,"num_write_kb":28672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111171+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:42:13.111171+0000","last_peered":"2024-09-11T14:42:13.111171+0000","last_clean":"2024-09-11T14:42:13.111171+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:42:13.111171+0000","last_undegraded":"2024-09-11T14:42:13.111171+0000","last_fullsized":"2024-09-11T14:42:13.111171+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"27'461","reported_seq":497,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:35.839073+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:42:35.839073+0000","last_peered":"2024-09-11T14:42:35.839073+0000","last_clean":"2024-09-11T14:42:35.839073+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:42:35.839073+0000","last_undegraded":"2024-09-11T14:42:35.839073+0000","last_fullsized":"2024-09-11T14:42:35.839073+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30212096,"num_objects":461,"num_object_clones":0,"num_object_copies":1383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":461,"num_write_kb":29504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.181385+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:42:13.181385+0000","last_peered":"2024-09-11T14:42:13.181385+0000","last_clean":"2024-09-11T14:42:13.181385+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:42:13.181385+0000","last_undegraded":"2024-09-11T14:42:13.181385+0000","last_fullsized":"2024-09-11T14:42:13.181385+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":461307904,"num_objects":7039,"num_object_clones":0,"num_object_copies":21117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7039,"num_write_kb":450496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":211417047040,"internally_reserved":0,"allocated":2794446848,"data_stored":2794446848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7039,"ondisk_log_size":7039,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":211417309184,"internally_reserved":0,"allocated":2794184704,"data_stored":2794184704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":52748988416,"internally_reserved":0,"allocated":803885056,"data_stored":803885056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673049,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":632396,"kb_used_data":632396,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51665332,"statfs":{"total":53552873472,"available":52905299968,"internally_reserved":0,"allocated":647573504,"data_stored":647573504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705753,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":776292,"kb_used_data":776292,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51521436,"statfs":{"total":53552873472,"available":52757950464,"internally_reserved":0,"allocated":794923008,"data_stored":794923008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705753,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":676468,"kb_used_data":676468,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51621260,"statfs":{"total":53552873472,"available":52860170240,"internally_reserved":0,"allocated":692703232,"data_stored":692703232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705753,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":614212,"kb_used_data":614212,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51683516,"statfs":{"total":53552873472,"available":52923920384,"internally_reserved":0,"allocated":628953088,"data_stored":628953088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":52748988416,"internally_reserved":0,"allocated":803885056,"data_stored":803885056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":52917444608,"internally_reserved":0,"allocated":635428864,"data_stored":635428864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":52852027392,"internally_reserved":0,"allocated":700846080,"data_stored":700846080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":52898848768,"internally_reserved":0,"allocated":654024704,"data_stored":654024704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":52748988416,"internally_reserved":0,"allocated":803885056,"data_stored":803885056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":52917444608,"internally_reserved":0,"allocated":635428864,"data_stored":635428864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":52852027392,"internally_reserved":0,"allocated":700846080,"data_stored":700846080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":52898586624,"internally_reserved":0,"allocated":654286848,"data_stored":654286848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":52748988416,"internally_reserved":0,"allocated":803885056,"data_stored":803885056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:42:37.911 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 77 16 7294 7278 5.90688 6.3125 0.370305 0.169101 2024-09-11T14:42:38.879 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:42:38.911 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 78 16 7377 7361 5.89765 5.1875 0.190146 0.169217 2024-09-11T14:42:39.025 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:39.020+0000 7fb6ea8c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:39.030 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:39.029+0000 7fb6ea8c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:39.911 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 79 16 7469 7453 5.89577 5.75 0.164278 0.169441 2024-09-11T14:42:40.911 INFO:tasks.radosbench.radosbench.0.smithi181.stdout:2024-09-11T14:42:40.911045+0000 min lat: 0.0249053 max lat: 1.01958 avg lat: 0.169744 2024-09-11T14:42:40.911 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T14:42:40.911 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 80 16 7548 7532 5.88378 4.9375 0.237093 0.169744 2024-09-11T14:42:41.612 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:42:41.756 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:41.752+0000 7f6bd3908640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:41.762 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:41.761+0000 7f6bd3908640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:41.911 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 81 16 7629 7613 5.87364 5.0625 0.0970985 0.170006 2024-09-11T14:42:42.037 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:42:42.037 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:42:42.037 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:42:42.193 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:42.184+0000 7f315829c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:42.201 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:42.199+0000 7f315829c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:42.472 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:42:42.472 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:42:42.493 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2024-09-11T14:42:40.724812+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":484311603,"num_objects":7385,"num_object_clones":0,"num_object_copies":22155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":7462,"num_write_kb":473492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7436,"ondisk_log_size":7436,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":2829480,"kb_used_data":2829480,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":206361432,"statfs":{"total":214211493888,"available":211314106368,"internally_reserved":0,"allocated":2897387520,"data_stored":2897387520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":81788928,"num_objects":1248,"num_object_clones":0,"num_object_copies":3744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1248,"num_write_kb":79872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002730"},"pg_stats":[{"pgid":"3.f","version":"27'445","reported_seq":481,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:39.549892+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:42:39.549892+0000","last_peered":"2024-09-11T14:42:39.549892+0000","last_clean":"2024-09-11T14:42:39.549892+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:42:39.549892+0000","last_undegraded":"2024-09-11T14:42:39.549892+0000","last_fullsized":"2024-09-11T14:42:39.549892+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":445,"log_dups_size":0,"ondisk_log_size":445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29163520,"num_objects":445,"num_object_clones":0,"num_object_copies":1335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":445,"num_write_kb":28480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"27'495","reported_seq":531,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:35.445372+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:42:35.445372+0000","last_peered":"2024-09-11T14:42:35.445372+0000","last_clean":"2024-09-11T14:42:35.445372+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:42:35.445372+0000","last_undegraded":"2024-09-11T14:42:35.445372+0000","last_fullsized":"2024-09-11T14:42:35.445372+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":495,"log_dups_size":0,"ondisk_log_size":495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32440320,"num_objects":495,"num_object_clones":0,"num_object_copies":1485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":495,"num_write_kb":31680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"27'460","reported_seq":496,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:39.643566+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:42:39.643566+0000","last_peered":"2024-09-11T14:42:39.643566+0000","last_clean":"2024-09-11T14:42:39.643566+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:42:39.643566+0000","last_undegraded":"2024-09-11T14:42:39.643566+0000","last_fullsized":"2024-09-11T14:42:39.643566+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30146560,"num_objects":460,"num_object_clones":0,"num_object_copies":1380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":460,"num_write_kb":29440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"27'470","reported_seq":506,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:39.434491+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:42:39.434491+0000","last_peered":"2024-09-11T14:42:39.434491+0000","last_clean":"2024-09-11T14:42:39.434491+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:42:39.434491+0000","last_undegraded":"2024-09-11T14:42:39.434491+0000","last_fullsized":"2024-09-11T14:42:39.434491+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":470,"log_dups_size":0,"ondisk_log_size":470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30801920,"num_objects":470,"num_object_clones":0,"num_object_copies":1410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":470,"num_write_kb":30080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"27'459","reported_seq":495,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:39.617422+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:42:39.617422+0000","last_peered":"2024-09-11T14:42:39.617422+0000","last_clean":"2024-09-11T14:42:39.617422+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:42:39.617422+0000","last_undegraded":"2024-09-11T14:42:39.617422+0000","last_fullsized":"2024-09-11T14:42:39.617422+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":459,"log_dups_size":0,"ondisk_log_size":459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30081024,"num_objects":459,"num_object_clones":0,"num_object_copies":1377,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":459,"num_write_kb":29376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"27'511","reported_seq":547,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:39.186524+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:42:39.186524+0000","last_peered":"2024-09-11T14:42:39.186524+0000","last_clean":"2024-09-11T14:42:39.186524+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:42:39.186524+0000","last_undegraded":"2024-09-11T14:42:39.186524+0000","last_fullsized":"2024-09-11T14:42:39.186524+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":511,"log_dups_size":0,"ondisk_log_size":511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33488896,"num_objects":511,"num_object_clones":0,"num_object_copies":1533,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":511,"num_write_kb":32704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"27'482","reported_seq":518,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:39.297041+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:42:39.297041+0000","last_peered":"2024-09-11T14:42:39.297041+0000","last_clean":"2024-09-11T14:42:39.297041+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:42:39.297041+0000","last_undegraded":"2024-09-11T14:42:39.297041+0000","last_fullsized":"2024-09-11T14:42:39.297041+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":482,"log_dups_size":0,"ondisk_log_size":482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31588352,"num_objects":482,"num_object_clones":0,"num_object_copies":1446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":482,"num_write_kb":30848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"27'444","reported_seq":480,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:39.445797+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:42:39.445797+0000","last_peered":"2024-09-11T14:42:39.445797+0000","last_clean":"2024-09-11T14:42:39.445797+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:42:39.445797+0000","last_undegraded":"2024-09-11T14:42:39.445797+0000","last_fullsized":"2024-09-11T14:42:39.445797+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29097984,"num_objects":444,"num_object_clones":0,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":444,"num_write_kb":28416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":121,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084543+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:42:13.084543+0000","last_peered":"2024-09-11T14:42:13.084543+0000","last_clean":"2024-09-11T14:42:13.084543+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:42:13.084543+0000","last_undegraded":"2024-09-11T14:42:13.084543+0000","last_fullsized":"2024-09-11T14:42:13.084543+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":50,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.184283+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:42:13.184283+0000","last_peered":"2024-09-11T14:42:13.184283+0000","last_clean":"2024-09-11T14:42:13.184283+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:42:13.184283+0000","last_undegraded":"2024-09-11T14:42:13.184283+0000","last_fullsized":"2024-09-11T14:42:13.184283+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"27'457","reported_seq":493,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:39.672616+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:42:39.672616+0000","last_peered":"2024-09-11T14:42:39.672616+0000","last_clean":"2024-09-11T14:42:39.672616+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:42:39.672616+0000","last_undegraded":"2024-09-11T14:42:39.672616+0000","last_fullsized":"2024-09-11T14:42:39.672616+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":457,"log_dups_size":0,"ondisk_log_size":457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29949952,"num_objects":457,"num_object_clones":0,"num_object_copies":1371,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":457,"num_write_kb":29248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084333+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:42:13.084333+0000","last_peered":"2024-09-11T14:42:13.084333+0000","last_clean":"2024-09-11T14:42:13.084333+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:42:13.084333+0000","last_undegraded":"2024-09-11T14:42:13.084333+0000","last_fullsized":"2024-09-11T14:42:13.084333+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"27'446","reported_seq":482,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:39.716370+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:42:39.716370+0000","last_peered":"2024-09-11T14:42:39.716370+0000","last_clean":"2024-09-11T14:42:39.716370+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:42:39.716370+0000","last_undegraded":"2024-09-11T14:42:39.716370+0000","last_fullsized":"2024-09-11T14:42:39.716370+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":446,"log_dups_size":0,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29229056,"num_objects":446,"num_object_clones":0,"num_object_copies":1338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":446,"num_write_kb":28544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.179633+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:42:13.179633+0000","last_peered":"2024-09-11T14:42:13.179633+0000","last_clean":"2024-09-11T14:42:13.179633+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:42:13.179633+0000","last_undegraded":"2024-09-11T14:42:13.179633+0000","last_fullsized":"2024-09-11T14:42:13.179633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"27'456","reported_seq":492,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:39.650655+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:42:39.650655+0000","last_peered":"2024-09-11T14:42:39.650655+0000","last_clean":"2024-09-11T14:42:39.650655+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:42:39.650655+0000","last_undegraded":"2024-09-11T14:42:39.650655+0000","last_fullsized":"2024-09-11T14:42:39.650655+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":456,"log_dups_size":0,"ondisk_log_size":456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29884416,"num_objects":456,"num_object_clones":0,"num_object_copies":1368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":456,"num_write_kb":29184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":51,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.082423+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:42:13.082423+0000","last_peered":"2024-09-11T14:42:13.082423+0000","last_clean":"2024-09-11T14:42:13.082423+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:42:13.082423+0000","last_undegraded":"2024-09-11T14:42:13.082423+0000","last_fullsized":"2024-09-11T14:42:13.082423+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"27'429","reported_seq":465,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:35.673482+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:42:35.673482+0000","last_peered":"2024-09-11T14:42:35.673482+0000","last_clean":"2024-09-11T14:42:35.673482+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:42:35.673482+0000","last_undegraded":"2024-09-11T14:42:35.673482+0000","last_fullsized":"2024-09-11T14:42:35.673482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":429,"log_dups_size":0,"ondisk_log_size":429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28114944,"num_objects":429,"num_object_clones":0,"num_object_copies":1287,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":429,"num_write_kb":27456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"27'467","reported_seq":503,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:39.390574+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:42:39.390574+0000","last_peered":"2024-09-11T14:42:39.390574+0000","last_clean":"2024-09-11T14:42:39.390574+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:42:39.390574+0000","last_undegraded":"2024-09-11T14:42:39.390574+0000","last_fullsized":"2024-09-11T14:42:39.390574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":467,"log_dups_size":0,"ondisk_log_size":467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30605312,"num_objects":467,"num_object_clones":0,"num_object_copies":1401,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":467,"num_write_kb":29888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111507+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:42:13.111507+0000","last_peered":"2024-09-11T14:42:13.111507+0000","last_clean":"2024-09-11T14:42:13.111507+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:42:13.111507+0000","last_undegraded":"2024-09-11T14:42:13.111507+0000","last_fullsized":"2024-09-11T14:42:13.111507+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"27'429","reported_seq":465,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:35.635112+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:42:35.635112+0000","last_peered":"2024-09-11T14:42:35.635112+0000","last_clean":"2024-09-11T14:42:35.635112+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:42:35.635112+0000","last_undegraded":"2024-09-11T14:42:35.635112+0000","last_fullsized":"2024-09-11T14:42:35.635112+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":429,"log_dups_size":0,"ondisk_log_size":429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28114944,"num_objects":429,"num_object_clones":0,"num_object_copies":1287,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":429,"num_write_kb":27456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.086961+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:42:13.086961+0000","last_peered":"2024-09-11T14:42:13.086961+0000","last_clean":"2024-09-11T14:42:13.086961+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:42:13.086961+0000","last_undegraded":"2024-09-11T14:42:13.086961+0000","last_fullsized":"2024-09-11T14:42:13.086961+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"27'470","reported_seq":506,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:39.493608+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:42:39.493608+0000","last_peered":"2024-09-11T14:42:39.493608+0000","last_clean":"2024-09-11T14:42:39.493608+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:42:39.493608+0000","last_undegraded":"2024-09-11T14:42:39.493608+0000","last_fullsized":"2024-09-11T14:42:39.493608+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":470,"log_dups_size":0,"ondisk_log_size":470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30801920,"num_objects":470,"num_object_clones":0,"num_object_copies":1410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":470,"num_write_kb":30080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111171+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:42:13.111171+0000","last_peered":"2024-09-11T14:42:13.111171+0000","last_clean":"2024-09-11T14:42:13.111171+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:42:13.111171+0000","last_undegraded":"2024-09-11T14:42:13.111171+0000","last_fullsized":"2024-09-11T14:42:13.111171+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"27'461","reported_seq":497,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:35.839073+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:42:35.839073+0000","last_peered":"2024-09-11T14:42:35.839073+0000","last_clean":"2024-09-11T14:42:35.839073+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:42:35.839073+0000","last_undegraded":"2024-09-11T14:42:35.839073+0000","last_fullsized":"2024-09-11T14:42:35.839073+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30212096,"num_objects":461,"num_object_clones":0,"num_object_copies":1383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":461,"num_write_kb":29504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.181385+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:42:13.181385+0000","last_peered":"2024-09-11T14:42:13.181385+0000","last_clean":"2024-09-11T14:42:13.181385+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:42:13.181385+0000","last_undegraded":"2024-09-11T14:42:13.181385+0000","last_fullsized":"2024-09-11T14:42:13.181385+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":483721216,"num_objects":7381,"num_object_clones":0,"num_object_copies":22143,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7381,"num_write_kb":472384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":211281297408,"internally_reserved":0,"allocated":2930196480,"data_stored":2930196480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7381,"ondisk_log_size":7381,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":211281690624,"internally_reserved":0,"allocated":2929803264,"data_stored":2929803264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":52697001984,"internally_reserved":0,"allocated":855871488,"data_stored":855871488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673049,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":632396,"kb_used_data":632396,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51665332,"statfs":{"total":53552873472,"available":52905299968,"internally_reserved":0,"allocated":647573504,"data_stored":647573504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705758,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":825492,"kb_used_data":825492,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51472236,"statfs":{"total":53552873472,"available":52707569664,"internally_reserved":0,"allocated":845303808,"data_stored":845303808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705758,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":719780,"kb_used_data":719780,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51577948,"statfs":{"total":53552873472,"available":52815818752,"internally_reserved":0,"allocated":737054720,"data_stored":737054720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705758,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":651812,"kb_used_data":651812,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51645916,"statfs":{"total":53552873472,"available":52885417984,"internally_reserved":0,"allocated":667455488,"data_stored":667455488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":52697001984,"internally_reserved":0,"allocated":855871488,"data_stored":855871488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":52878290944,"internally_reserved":0,"allocated":674582528,"data_stored":674582528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":52807548928,"internally_reserved":0,"allocated":745324544,"data_stored":745324544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":52898848768,"internally_reserved":0,"allocated":654024704,"data_stored":654024704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":52697001984,"internally_reserved":0,"allocated":855871488,"data_stored":855871488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":52878290944,"internally_reserved":0,"allocated":674582528,"data_stored":674582528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":52807417856,"internally_reserved":0,"allocated":745455616,"data_stored":745455616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":52898586624,"internally_reserved":0,"allocated":654286848,"data_stored":654286848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":52697001984,"internally_reserved":0,"allocated":855871488,"data_stored":855871488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:42:42.911 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 82 16 7710 7694 5.86374 5.0625 0.132914 0.170278 2024-09-11T14:42:43.911 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 83 16 7791 7775 5.85408 5.0625 0.0548814 0.170582 2024-09-11T14:42:44.143 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:42:44.296 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:44.292+0000 7f8efc591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:44.301 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:44.299+0000 7f8efc591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:44.911 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 84 16 7879 7863 5.84986 5.5 0.17897 0.170644 2024-09-11T14:42:45.911 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 85 16 7960 7944 5.84059 5.0625 0.310665 0.170835 2024-09-11T14:42:46.879 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:42:46.911 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 86 16 8042 8026 5.83227 5.125 0.165045 0.171244 2024-09-11T14:42:47.024 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:47.021+0000 7f7cbe751640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:47.029 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:47.027+0000 7f7cbe751640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:47.496 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:42:47.496 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:42:47.497 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:42:47.662 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:47.655+0000 7fee69f11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:47.667 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:47.666+0000 7fee69f11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:47.912 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 87 16 8113 8097 5.81623 4.4375 0.8376 0.171511 2024-09-11T14:42:48.041 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:42:48.042 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:42:48.060 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2024-09-11T14:42:46.726133+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":517931571,"num_objects":7898,"num_object_clones":0,"num_object_copies":23694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":7975,"num_write_kb":506324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7949,"ondisk_log_size":7949,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":3029124,"kb_used_data":3029124,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":206161788,"statfs":{"total":214211493888,"available":211109670912,"internally_reserved":0,"allocated":3101822976,"data_stored":3101822976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":85721088,"num_objects":1308,"num_object_clones":0,"num_object_copies":3924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1308,"num_write_kb":83712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002694"},"pg_stats":[{"pgid":"3.f","version":"27'477","reported_seq":513,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:44.750671+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:42:44.750671+0000","last_peered":"2024-09-11T14:42:44.750671+0000","last_clean":"2024-09-11T14:42:44.750671+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:42:44.750671+0000","last_undegraded":"2024-09-11T14:42:44.750671+0000","last_fullsized":"2024-09-11T14:42:44.750671+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31260672,"num_objects":477,"num_object_clones":0,"num_object_copies":1431,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":477,"num_write_kb":30528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"27'541","reported_seq":577,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:45.614135+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:42:45.614135+0000","last_peered":"2024-09-11T14:42:45.614135+0000","last_clean":"2024-09-11T14:42:45.614135+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:42:45.614135+0000","last_undegraded":"2024-09-11T14:42:45.614135+0000","last_fullsized":"2024-09-11T14:42:45.614135+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":541,"log_dups_size":0,"ondisk_log_size":541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35454976,"num_objects":541,"num_object_clones":0,"num_object_copies":1623,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":541,"num_write_kb":34624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"27'494","reported_seq":530,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:44.709803+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:42:44.709803+0000","last_peered":"2024-09-11T14:42:44.709803+0000","last_clean":"2024-09-11T14:42:44.709803+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:42:44.709803+0000","last_undegraded":"2024-09-11T14:42:44.709803+0000","last_fullsized":"2024-09-11T14:42:44.709803+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":494,"log_dups_size":0,"ondisk_log_size":494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32374784,"num_objects":494,"num_object_clones":0,"num_object_copies":1482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":494,"num_write_kb":31616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"27'494","reported_seq":530,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:44.476670+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:42:44.476670+0000","last_peered":"2024-09-11T14:42:44.476670+0000","last_clean":"2024-09-11T14:42:44.476670+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:42:44.476670+0000","last_undegraded":"2024-09-11T14:42:44.476670+0000","last_fullsized":"2024-09-11T14:42:44.476670+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":494,"log_dups_size":0,"ondisk_log_size":494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32374784,"num_objects":494,"num_object_clones":0,"num_object_copies":1482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":494,"num_write_kb":31616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"27'487","reported_seq":523,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:44.105113+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:42:44.105113+0000","last_peered":"2024-09-11T14:42:44.105113+0000","last_clean":"2024-09-11T14:42:44.105113+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:42:44.105113+0000","last_undegraded":"2024-09-11T14:42:44.105113+0000","last_fullsized":"2024-09-11T14:42:44.105113+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31916032,"num_objects":487,"num_object_clones":0,"num_object_copies":1461,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":487,"num_write_kb":31168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"27'538","reported_seq":574,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:44.545661+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:42:44.545661+0000","last_peered":"2024-09-11T14:42:44.545661+0000","last_clean":"2024-09-11T14:42:44.545661+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:42:44.545661+0000","last_undegraded":"2024-09-11T14:42:44.545661+0000","last_fullsized":"2024-09-11T14:42:44.545661+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":538,"log_dups_size":0,"ondisk_log_size":538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35258368,"num_objects":538,"num_object_clones":0,"num_object_copies":1614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":538,"num_write_kb":34432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"27'505","reported_seq":541,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:44.606057+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:42:44.606057+0000","last_peered":"2024-09-11T14:42:44.606057+0000","last_clean":"2024-09-11T14:42:44.606057+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:42:44.606057+0000","last_undegraded":"2024-09-11T14:42:44.606057+0000","last_fullsized":"2024-09-11T14:42:44.606057+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":505,"log_dups_size":0,"ondisk_log_size":505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33095680,"num_objects":505,"num_object_clones":0,"num_object_copies":1515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":505,"num_write_kb":32320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"27'473","reported_seq":509,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:44.134151+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:42:44.134151+0000","last_peered":"2024-09-11T14:42:44.134151+0000","last_clean":"2024-09-11T14:42:44.134151+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:42:44.134151+0000","last_undegraded":"2024-09-11T14:42:44.134151+0000","last_fullsized":"2024-09-11T14:42:44.134151+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":473,"log_dups_size":0,"ondisk_log_size":473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30998528,"num_objects":473,"num_object_clones":0,"num_object_copies":1419,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":473,"num_write_kb":30272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":121,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084543+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:42:13.084543+0000","last_peered":"2024-09-11T14:42:13.084543+0000","last_clean":"2024-09-11T14:42:13.084543+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:42:13.084543+0000","last_undegraded":"2024-09-11T14:42:13.084543+0000","last_fullsized":"2024-09-11T14:42:13.084543+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":50,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.184283+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:42:13.184283+0000","last_peered":"2024-09-11T14:42:13.184283+0000","last_clean":"2024-09-11T14:42:13.184283+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:42:13.184283+0000","last_undegraded":"2024-09-11T14:42:13.184283+0000","last_fullsized":"2024-09-11T14:42:13.184283+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"27'483","reported_seq":519,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:44.340154+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:42:44.340154+0000","last_peered":"2024-09-11T14:42:44.340154+0000","last_clean":"2024-09-11T14:42:44.340154+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:42:44.340154+0000","last_undegraded":"2024-09-11T14:42:44.340154+0000","last_fullsized":"2024-09-11T14:42:44.340154+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":483,"log_dups_size":0,"ondisk_log_size":483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31653888,"num_objects":483,"num_object_clones":0,"num_object_copies":1449,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":483,"num_write_kb":30912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084333+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:42:13.084333+0000","last_peered":"2024-09-11T14:42:13.084333+0000","last_clean":"2024-09-11T14:42:13.084333+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:42:13.084333+0000","last_undegraded":"2024-09-11T14:42:13.084333+0000","last_fullsized":"2024-09-11T14:42:13.084333+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"27'472","reported_seq":508,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:44.494224+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:42:44.494224+0000","last_peered":"2024-09-11T14:42:44.494224+0000","last_clean":"2024-09-11T14:42:44.494224+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:42:44.494224+0000","last_undegraded":"2024-09-11T14:42:44.494224+0000","last_fullsized":"2024-09-11T14:42:44.494224+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":472,"log_dups_size":0,"ondisk_log_size":472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30932992,"num_objects":472,"num_object_clones":0,"num_object_copies":1416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":472,"num_write_kb":30208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.179633+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:42:13.179633+0000","last_peered":"2024-09-11T14:42:13.179633+0000","last_clean":"2024-09-11T14:42:13.179633+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:42:13.179633+0000","last_undegraded":"2024-09-11T14:42:13.179633+0000","last_fullsized":"2024-09-11T14:42:13.179633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"27'484","reported_seq":520,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:44.777669+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:42:44.777669+0000","last_peered":"2024-09-11T14:42:44.777669+0000","last_clean":"2024-09-11T14:42:44.777669+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:42:44.777669+0000","last_undegraded":"2024-09-11T14:42:44.777669+0000","last_fullsized":"2024-09-11T14:42:44.777669+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":484,"log_dups_size":0,"ondisk_log_size":484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31719424,"num_objects":484,"num_object_clones":0,"num_object_copies":1452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":484,"num_write_kb":30976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":51,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.082423+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:42:13.082423+0000","last_peered":"2024-09-11T14:42:13.082423+0000","last_clean":"2024-09-11T14:42:13.082423+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:42:13.082423+0000","last_undegraded":"2024-09-11T14:42:13.082423+0000","last_fullsized":"2024-09-11T14:42:13.082423+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"27'479","reported_seq":515,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:45.326421+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:42:45.326421+0000","last_peered":"2024-09-11T14:42:45.326421+0000","last_clean":"2024-09-11T14:42:45.326421+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:42:45.326421+0000","last_undegraded":"2024-09-11T14:42:45.326421+0000","last_fullsized":"2024-09-11T14:42:45.326421+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":479,"log_dups_size":0,"ondisk_log_size":479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31391744,"num_objects":479,"num_object_clones":0,"num_object_copies":1437,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":479,"num_write_kb":30656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"27'486","reported_seq":522,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:44.433853+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:42:44.433853+0000","last_peered":"2024-09-11T14:42:44.433853+0000","last_clean":"2024-09-11T14:42:44.433853+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:42:44.433853+0000","last_undegraded":"2024-09-11T14:42:44.433853+0000","last_fullsized":"2024-09-11T14:42:44.433853+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":486,"log_dups_size":0,"ondisk_log_size":486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31850496,"num_objects":486,"num_object_clones":0,"num_object_copies":1458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":486,"num_write_kb":31104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111507+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:42:13.111507+0000","last_peered":"2024-09-11T14:42:13.111507+0000","last_clean":"2024-09-11T14:42:13.111507+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:42:13.111507+0000","last_undegraded":"2024-09-11T14:42:13.111507+0000","last_fullsized":"2024-09-11T14:42:13.111507+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"27'467","reported_seq":503,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:45.286547+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:42:45.286547+0000","last_peered":"2024-09-11T14:42:45.286547+0000","last_clean":"2024-09-11T14:42:45.286547+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:42:45.286547+0000","last_undegraded":"2024-09-11T14:42:45.286547+0000","last_fullsized":"2024-09-11T14:42:45.286547+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":467,"log_dups_size":0,"ondisk_log_size":467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30605312,"num_objects":467,"num_object_clones":0,"num_object_copies":1401,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":467,"num_write_kb":29888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.086961+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:42:13.086961+0000","last_peered":"2024-09-11T14:42:13.086961+0000","last_clean":"2024-09-11T14:42:13.086961+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:42:13.086961+0000","last_undegraded":"2024-09-11T14:42:13.086961+0000","last_fullsized":"2024-09-11T14:42:13.086961+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"27'497","reported_seq":533,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:44.627559+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:42:44.627559+0000","last_peered":"2024-09-11T14:42:44.627559+0000","last_clean":"2024-09-11T14:42:44.627559+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:42:44.627559+0000","last_undegraded":"2024-09-11T14:42:44.627559+0000","last_fullsized":"2024-09-11T14:42:44.627559+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":497,"log_dups_size":0,"ondisk_log_size":497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32571392,"num_objects":497,"num_object_clones":0,"num_object_copies":1491,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":497,"num_write_kb":31808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111171+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:42:13.111171+0000","last_peered":"2024-09-11T14:42:13.111171+0000","last_clean":"2024-09-11T14:42:13.111171+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:42:13.111171+0000","last_undegraded":"2024-09-11T14:42:13.111171+0000","last_fullsized":"2024-09-11T14:42:13.111171+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"27'517","reported_seq":553,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:45.382318+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:42:45.382318+0000","last_peered":"2024-09-11T14:42:45.382318+0000","last_clean":"2024-09-11T14:42:45.382318+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:42:45.382318+0000","last_undegraded":"2024-09-11T14:42:45.382318+0000","last_fullsized":"2024-09-11T14:42:45.382318+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":517,"log_dups_size":0,"ondisk_log_size":517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33882112,"num_objects":517,"num_object_clones":0,"num_object_copies":1551,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":517,"num_write_kb":33088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.181385+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:42:13.181385+0000","last_peered":"2024-09-11T14:42:13.181385+0000","last_clean":"2024-09-11T14:42:13.181385+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:42:13.181385+0000","last_undegraded":"2024-09-11T14:42:13.181385+0000","last_fullsized":"2024-09-11T14:42:13.181385+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":517341184,"num_objects":7894,"num_object_clones":0,"num_object_copies":23682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7894,"num_write_kb":505216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":211077738496,"internally_reserved":0,"allocated":3133755392,"data_stored":3133755392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7894,"ondisk_log_size":7894,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":211078004736,"internally_reserved":0,"allocated":3133489152,"data_stored":3133489152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":52649037824,"internally_reserved":0,"allocated":903835648,"data_stored":903835648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673059,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":706460,"kb_used_data":706460,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51591268,"statfs":{"total":53552873472,"available":52829458432,"internally_reserved":0,"allocated":723415040,"data_stored":723415040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705763,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":873096,"kb_used_data":873096,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51424632,"statfs":{"total":53552873472,"available":52658823168,"internally_reserved":0,"allocated":894050304,"data_stored":894050304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705763,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":760440,"kb_used_data":760440,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51537288,"statfs":{"total":53552873472,"available":52774182912,"internally_reserved":0,"allocated":778690560,"data_stored":778690560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705763,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":689128,"kb_used_data":689128,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51608600,"statfs":{"total":53552873472,"available":52847206400,"internally_reserved":0,"allocated":705667072,"data_stored":705667072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":52649037824,"internally_reserved":0,"allocated":903835648,"data_stored":903835648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":52840042496,"internally_reserved":0,"allocated":712830976,"data_stored":712830976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":52765405184,"internally_reserved":0,"allocated":787468288,"data_stored":787468288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":52823650304,"internally_reserved":0,"allocated":729223168,"data_stored":729223168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":52648906752,"internally_reserved":0,"allocated":903966720,"data_stored":903966720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":52840042496,"internally_reserved":0,"allocated":712830976,"data_stored":712830976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":52765405184,"internally_reserved":0,"allocated":787468288,"data_stored":787468288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":52823515136,"internally_reserved":0,"allocated":729358336,"data_stored":729358336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":52648775680,"internally_reserved":0,"allocated":904097792,"data_stored":904097792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:42:48.912 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 88 16 8184 8168 5.80056 4.4375 0.105583 0.172013 2024-09-11T14:42:49.447 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:42:49.620 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:49.616+0000 7fd067191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:49.627 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:49.625+0000 7fd067191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:49.912 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 89 16 8269 8253 5.79507 5.3125 0.0833729 0.172362 2024-09-11T14:42:50.912 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 90 16 8337 8321 5.7779 4.25 0.251529 0.172704 2024-09-11T14:42:51.912 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 91 16 8425 8409 5.77484 5.5 0.227568 0.172973 2024-09-11T14:42:52.087 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:42:52.241 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:52.238+0000 7f58272ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:52.246 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:52.244+0000 7f58272ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:52.912 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 92 16 8499 8483 5.76234 4.625 0.32661 0.173075 2024-09-11T14:42:53.062 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:42:53.063 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:42:53.063 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-11T14:42:53.063 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-11T14:42:53.240 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:53.228+0000 7f86c1463640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:53.244 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:53.243+0000 7f86c1463640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:53.321 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:42:53.321 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:42:53.322 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:42:53.322 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:42:53.322 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:42:53.322 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:42:53.322 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:42:53.322 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:42:53.322 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:42:53.322 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:42:53.322 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:42:53.322 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:42:53.326 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:42:53.326 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:42:53.327 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-11T14:42:53.480 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:53.468+0000 7f114f0fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:53.488 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:53.486+0000 7f114f0fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:53.681 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:42:53.681 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:42:53.681 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:42:53.681 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:42:53.681 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:42:53.681 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:42:53.681 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:42:53.681 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:42:53.682 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:42:53.682 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:42:53.682 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:42:53.682 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:42:53.686 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:42:53.687 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:42:53.687 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-11T14:42:53.845 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:53.833+0000 7fe1a8391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:53.855 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:53.851+0000 7fe1a8391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:53.912 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 93 16 8591 8575 5.7622 5.75 0.152688 0.173348 2024-09-11T14:42:53.985 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:42:53.985 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:42:53.985 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:42:53.985 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:42:53.985 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:42:53.985 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:42:53.986 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:42:53.986 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:42:53.986 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:42:53.986 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:42:53.986 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:42:53.986 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:42:53.990 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:42:53.991 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:42:53.991 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-11T14:42:54.173 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:54.165+0000 7f9105f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:54.178 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:54.177+0000 7f9105f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:54.319 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:42:54.319 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:42:54.319 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:42:54.319 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:42:54.320 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:42:54.320 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:42:54.320 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:42:54.320 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:42:54.320 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:42:54.320 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:42:54.320 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:42:54.320 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:42:54.324 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:42:54.324 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:42:54.581 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:42:54.741 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:54.737+0000 7ff3c90c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:54.748 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:54.746+0000 7ff3c90c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:54.912 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 94 16 8668 8652 5.75209 4.8125 0.340673 0.173629 2024-09-11T14:42:55.912 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 95 16 8756 8740 5.74943 5.5 0.0913191 0.173655 2024-09-11T14:42:56.913 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 96 16 8837 8821 5.74227 5.0625 0.17273 0.173938 2024-09-11T14:42:57.446 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:42:57.578 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:57.573+0000 7f20a5f51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:57.584 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:57.583+0000 7f20a5f51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:57.913 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 97 16 8919 8903 5.7359 5.125 0.158223 0.174117 2024-09-11T14:42:58.913 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 98 16 9002 8986 5.7303 5.1875 0.110077 0.174307 2024-09-11T14:42:59.325 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:42:59.325 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:42:59.325 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:42:59.485 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:59.478+0000 7fde4de9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:59.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:42:59.489+0000 7fde4de9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:59.863 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:42:59.864 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:42:59.888 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2024-09-11T14:42:58.728933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":569639475,"num_objects":8687,"num_object_clones":0,"num_object_copies":26061,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":8764,"num_write_kb":556820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8738,"ondisk_log_size":8738,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":3338200,"kb_used_data":3338200,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":205852712,"statfs":{"total":214211493888,"available":210793177088,"internally_reserved":0,"allocated":3418316800,"data_stored":3418316800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":51707904,"num_objects":789,"num_object_clones":0,"num_object_copies":2367,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":789,"num_write_kb":50496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002800"},"pg_stats":[{"pgid":"3.f","version":"27'525","reported_seq":561,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:54.664794+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:42:54.664794+0000","last_peered":"2024-09-11T14:42:54.664794+0000","last_clean":"2024-09-11T14:42:54.664794+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:42:54.664794+0000","last_undegraded":"2024-09-11T14:42:54.664794+0000","last_fullsized":"2024-09-11T14:42:54.664794+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":525,"log_dups_size":0,"ondisk_log_size":525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34406400,"num_objects":525,"num_object_clones":0,"num_object_copies":1575,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":525,"num_write_kb":33600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"27'597","reported_seq":633,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:55.232308+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:42:55.232308+0000","last_peered":"2024-09-11T14:42:55.232308+0000","last_clean":"2024-09-11T14:42:55.232308+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:42:55.232308+0000","last_undegraded":"2024-09-11T14:42:55.232308+0000","last_fullsized":"2024-09-11T14:42:55.232308+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":597,"log_dups_size":0,"ondisk_log_size":597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39124992,"num_objects":597,"num_object_clones":0,"num_object_copies":1791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":597,"num_write_kb":38208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"27'555","reported_seq":591,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:54.753784+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:42:54.753784+0000","last_peered":"2024-09-11T14:42:54.753784+0000","last_clean":"2024-09-11T14:42:54.753784+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:42:54.753784+0000","last_undegraded":"2024-09-11T14:42:54.753784+0000","last_fullsized":"2024-09-11T14:42:54.753784+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":555,"log_dups_size":0,"ondisk_log_size":555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36372480,"num_objects":555,"num_object_clones":0,"num_object_copies":1665,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":555,"num_write_kb":35520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"27'539","reported_seq":575,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:54.690978+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:42:54.690978+0000","last_peered":"2024-09-11T14:42:54.690978+0000","last_clean":"2024-09-11T14:42:54.690978+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:42:54.690978+0000","last_undegraded":"2024-09-11T14:42:54.690978+0000","last_fullsized":"2024-09-11T14:42:54.690978+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":539,"log_dups_size":0,"ondisk_log_size":539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35323904,"num_objects":539,"num_object_clones":0,"num_object_copies":1617,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":539,"num_write_kb":34496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"27'530","reported_seq":566,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:53.442009+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:42:53.442009+0000","last_peered":"2024-09-11T14:42:53.442009+0000","last_clean":"2024-09-11T14:42:53.442009+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:42:53.442009+0000","last_undegraded":"2024-09-11T14:42:53.442009+0000","last_fullsized":"2024-09-11T14:42:53.442009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34734080,"num_objects":530,"num_object_clones":0,"num_object_copies":1590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":530,"num_write_kb":33920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"27'585","reported_seq":621,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:54.266355+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:42:54.266355+0000","last_peered":"2024-09-11T14:42:54.266355+0000","last_clean":"2024-09-11T14:42:54.266355+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:42:54.266355+0000","last_undegraded":"2024-09-11T14:42:54.266355+0000","last_fullsized":"2024-09-11T14:42:54.266355+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":585,"log_dups_size":0,"ondisk_log_size":585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38338560,"num_objects":585,"num_object_clones":0,"num_object_copies":1755,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":585,"num_write_kb":37440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"27'562","reported_seq":598,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:54.805423+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:42:54.805423+0000","last_peered":"2024-09-11T14:42:54.805423+0000","last_clean":"2024-09-11T14:42:54.805423+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:42:54.805423+0000","last_undegraded":"2024-09-11T14:42:54.805423+0000","last_fullsized":"2024-09-11T14:42:54.805423+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":562,"log_dups_size":0,"ondisk_log_size":562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36831232,"num_objects":562,"num_object_clones":0,"num_object_copies":1686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":562,"num_write_kb":35968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"27'521","reported_seq":557,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:53.936695+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:42:53.936695+0000","last_peered":"2024-09-11T14:42:53.936695+0000","last_clean":"2024-09-11T14:42:53.936695+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:42:53.936695+0000","last_undegraded":"2024-09-11T14:42:53.936695+0000","last_fullsized":"2024-09-11T14:42:53.936695+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":521,"log_dups_size":0,"ondisk_log_size":521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34144256,"num_objects":521,"num_object_clones":0,"num_object_copies":1563,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":521,"num_write_kb":33344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":121,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084543+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:42:13.084543+0000","last_peered":"2024-09-11T14:42:13.084543+0000","last_clean":"2024-09-11T14:42:13.084543+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:42:13.084543+0000","last_undegraded":"2024-09-11T14:42:13.084543+0000","last_fullsized":"2024-09-11T14:42:13.084543+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":50,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.184283+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:42:13.184283+0000","last_peered":"2024-09-11T14:42:13.184283+0000","last_clean":"2024-09-11T14:42:13.184283+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:42:13.184283+0000","last_undegraded":"2024-09-11T14:42:13.184283+0000","last_fullsized":"2024-09-11T14:42:13.184283+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"27'531","reported_seq":567,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:54.579228+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:42:54.579228+0000","last_peered":"2024-09-11T14:42:54.579228+0000","last_clean":"2024-09-11T14:42:54.579228+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:42:54.579228+0000","last_undegraded":"2024-09-11T14:42:54.579228+0000","last_fullsized":"2024-09-11T14:42:54.579228+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":531,"log_dups_size":0,"ondisk_log_size":531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34799616,"num_objects":531,"num_object_clones":0,"num_object_copies":1593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":531,"num_write_kb":33984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084333+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:42:13.084333+0000","last_peered":"2024-09-11T14:42:13.084333+0000","last_clean":"2024-09-11T14:42:13.084333+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:42:13.084333+0000","last_undegraded":"2024-09-11T14:42:13.084333+0000","last_fullsized":"2024-09-11T14:42:13.084333+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"27'521","reported_seq":557,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:54.585599+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:42:54.585599+0000","last_peered":"2024-09-11T14:42:54.585599+0000","last_clean":"2024-09-11T14:42:54.585599+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:42:54.585599+0000","last_undegraded":"2024-09-11T14:42:54.585599+0000","last_fullsized":"2024-09-11T14:42:54.585599+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":521,"log_dups_size":0,"ondisk_log_size":521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34144256,"num_objects":521,"num_object_clones":0,"num_object_copies":1563,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":521,"num_write_kb":33344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.179633+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:42:13.179633+0000","last_peered":"2024-09-11T14:42:13.179633+0000","last_clean":"2024-09-11T14:42:13.179633+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:42:13.179633+0000","last_undegraded":"2024-09-11T14:42:13.179633+0000","last_fullsized":"2024-09-11T14:42:13.179633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"27'537","reported_seq":573,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:54.565879+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:42:54.565879+0000","last_peered":"2024-09-11T14:42:54.565879+0000","last_clean":"2024-09-11T14:42:54.565879+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:42:54.565879+0000","last_undegraded":"2024-09-11T14:42:54.565879+0000","last_fullsized":"2024-09-11T14:42:54.565879+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":537,"log_dups_size":0,"ondisk_log_size":537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35192832,"num_objects":537,"num_object_clones":0,"num_object_copies":1611,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":537,"num_write_kb":34368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":51,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.082423+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:42:13.082423+0000","last_peered":"2024-09-11T14:42:13.082423+0000","last_clean":"2024-09-11T14:42:13.082423+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:42:13.082423+0000","last_undegraded":"2024-09-11T14:42:13.082423+0000","last_fullsized":"2024-09-11T14:42:13.082423+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"27'525","reported_seq":561,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:55.670994+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:42:55.670994+0000","last_peered":"2024-09-11T14:42:55.670994+0000","last_clean":"2024-09-11T14:42:55.670994+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:42:55.670994+0000","last_undegraded":"2024-09-11T14:42:55.670994+0000","last_fullsized":"2024-09-11T14:42:55.670994+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":525,"log_dups_size":0,"ondisk_log_size":525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34406400,"num_objects":525,"num_object_clones":0,"num_object_copies":1575,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":525,"num_write_kb":33600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"27'532","reported_seq":568,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:54.373486+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:42:54.373486+0000","last_peered":"2024-09-11T14:42:54.373486+0000","last_clean":"2024-09-11T14:42:54.373486+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:42:54.373486+0000","last_undegraded":"2024-09-11T14:42:54.373486+0000","last_fullsized":"2024-09-11T14:42:54.373486+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":532,"log_dups_size":0,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34865152,"num_objects":532,"num_object_clones":0,"num_object_copies":1596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":532,"num_write_kb":34048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111507+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:42:13.111507+0000","last_peered":"2024-09-11T14:42:13.111507+0000","last_clean":"2024-09-11T14:42:13.111507+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:42:13.111507+0000","last_undegraded":"2024-09-11T14:42:13.111507+0000","last_fullsized":"2024-09-11T14:42:13.111507+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"27'522","reported_seq":558,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:55.502519+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:42:55.502519+0000","last_peered":"2024-09-11T14:42:55.502519+0000","last_clean":"2024-09-11T14:42:55.502519+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:42:55.502519+0000","last_undegraded":"2024-09-11T14:42:55.502519+0000","last_fullsized":"2024-09-11T14:42:55.502519+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":522,"log_dups_size":0,"ondisk_log_size":522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34209792,"num_objects":522,"num_object_clones":0,"num_object_copies":1566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":522,"num_write_kb":33408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.086961+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:42:13.086961+0000","last_peered":"2024-09-11T14:42:13.086961+0000","last_clean":"2024-09-11T14:42:13.086961+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:42:13.086961+0000","last_undegraded":"2024-09-11T14:42:13.086961+0000","last_fullsized":"2024-09-11T14:42:13.086961+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"27'538","reported_seq":574,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:54.664140+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:42:54.664140+0000","last_peered":"2024-09-11T14:42:54.664140+0000","last_clean":"2024-09-11T14:42:54.664140+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:42:54.664140+0000","last_undegraded":"2024-09-11T14:42:54.664140+0000","last_fullsized":"2024-09-11T14:42:54.664140+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":538,"log_dups_size":0,"ondisk_log_size":538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35258368,"num_objects":538,"num_object_clones":0,"num_object_copies":1614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":538,"num_write_kb":34432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111171+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:42:13.111171+0000","last_peered":"2024-09-11T14:42:13.111171+0000","last_clean":"2024-09-11T14:42:13.111171+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:42:13.111171+0000","last_undegraded":"2024-09-11T14:42:13.111171+0000","last_fullsized":"2024-09-11T14:42:13.111171+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"27'563","reported_seq":599,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:55.578391+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:42:55.578391+0000","last_peered":"2024-09-11T14:42:55.578391+0000","last_clean":"2024-09-11T14:42:55.578391+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:42:55.578391+0000","last_undegraded":"2024-09-11T14:42:55.578391+0000","last_fullsized":"2024-09-11T14:42:55.578391+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":563,"log_dups_size":0,"ondisk_log_size":563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36896768,"num_objects":563,"num_object_clones":0,"num_object_copies":1689,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":563,"num_write_kb":36032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.181385+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:42:13.181385+0000","last_peered":"2024-09-11T14:42:13.181385+0000","last_clean":"2024-09-11T14:42:13.181385+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:42:13.181385+0000","last_undegraded":"2024-09-11T14:42:13.181385+0000","last_fullsized":"2024-09-11T14:42:13.181385+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":569049088,"num_objects":8683,"num_object_clones":0,"num_object_copies":26049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8683,"num_write_kb":555712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":210763194368,"internally_reserved":0,"allocated":3448299520,"data_stored":3448299520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8683,"ondisk_log_size":8683,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":210763849728,"internally_reserved":0,"allocated":3447644160,"data_stored":3447644160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":52557475840,"internally_reserved":0,"allocated":995397632,"data_stored":995397632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673069,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":776228,"kb_used_data":776228,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51521500,"statfs":{"total":53552873472,"available":52758016000,"internally_reserved":0,"allocated":794857472,"data_stored":794857472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705773,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":963720,"kb_used_data":963720,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51334008,"statfs":{"total":53552873472,"available":52566024192,"internally_reserved":0,"allocated":986849280,"data_stored":986849280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705773,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":839284,"kb_used_data":839284,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51458444,"statfs":{"total":53552873472,"available":52693446656,"internally_reserved":0,"allocated":859426816,"data_stored":859426816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705773,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":758968,"kb_used_data":758968,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51538760,"statfs":{"total":53552873472,"available":52775690240,"internally_reserved":0,"allocated":777183232,"data_stored":777183232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":52557475840,"internally_reserved":0,"allocated":995397632,"data_stored":995397632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":52769001472,"internally_reserved":0,"allocated":783872000,"data_stored":783872000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":52686319616,"internally_reserved":0,"allocated":866553856,"data_stored":866553856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":52751052800,"internally_reserved":0,"allocated":801820672,"data_stored":801820672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":52557475840,"internally_reserved":0,"allocated":995397632,"data_stored":995397632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":52769001472,"internally_reserved":0,"allocated":783872000,"data_stored":783872000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":52685795328,"internally_reserved":0,"allocated":867078144,"data_stored":867078144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":52750921728,"internally_reserved":0,"allocated":801951744,"data_stored":801951744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":52557475840,"internally_reserved":0,"allocated":995397632,"data_stored":995397632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:42:59.913 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 99 16 9081 9065 5.72229 4.9375 0.102152 0.174473 2024-09-11T14:43:00.022 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:43:00.182 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:00.178+0000 7f1885888640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:00.187 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:00.186+0000 7f1885888640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:00.913 INFO:tasks.radosbench.radosbench.0.smithi181.stdout:2024-09-11T14:43:00.912869+0000 min lat: 0.0249053 max lat: 1.01958 avg lat: 0.174618 2024-09-11T14:43:00.913 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T14:43:00.913 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 100 16 9164 9148 5.71694 5.1875 0.0959305 0.174618 2024-09-11T14:43:01.913 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 101 16 9242 9226 5.7086 4.875 0.210959 0.17487 2024-09-11T14:43:02.830 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:43:02.913 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 102 16 9326 9310 5.70409 5.25 0.174704 0.175103 2024-09-11T14:43:02.979 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:02.975+0000 7f5580cc8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:02.984 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:02.982+0000 7f5580cc8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:03.913 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 103 16 9402 9386 5.69483 4.75 0.104448 0.175364 2024-09-11T14:43:04.889 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:43:04.890 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:43:04.890 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-11T14:43:04.890 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-11T14:43:04.913 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 104 15 9478 9463 5.68634 4.8125 0.144073 0.175653 2024-09-11T14:43:05.022 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:05.013+0000 7f1feb791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:05.027 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:05.026+0000 7f1feb791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:05.108 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:43:05.108 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:43:05.108 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:43:05.108 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:43:05.109 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:43:05.109 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:43:05.109 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:43:05.109 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:43:05.109 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:43:05.109 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:43:05.109 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:43:05.109 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:43:05.112 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:43:05.112 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:43:05.113 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-11T14:43:05.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:05.319+0000 7fa9538fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:05.332 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:05.331+0000 7fa9538fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:05.411 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:43:05.411 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:43:05.411 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:43:05.411 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:43:05.411 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:43:05.411 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:43:05.412 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:43:05.412 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:43:05.412 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:43:05.412 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:43:05.412 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:43:05.412 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:43:05.414 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:43:05.415 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:43:05.415 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-11T14:43:05.472 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:43:05.576 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:05.563+0000 7f909e751640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:05.590 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:05.584+0000 7f909e751640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:05.628 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:05.625+0000 7fcd2e391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:05.634 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:05.632+0000 7fcd2e391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:05.645 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:43:05.645 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:43:05.645 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:43:05.645 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:43:05.645 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:43:05.645 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:43:05.645 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:43:05.645 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:43:05.645 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:43:05.646 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:43:05.646 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:43:05.646 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:43:05.649 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:43:05.650 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:43:05.650 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-11T14:43:05.789 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:05.781+0000 7f0ce64d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:05.795 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:05.794+0000 7f0ce64d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:05.913 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 105 16 9555 9539 5.67741 4.75 0.101781 0.175955 2024-09-11T14:43:05.965 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:43:05.965 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:43:05.965 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:43:05.966 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:43:05.966 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:43:05.966 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:43:05.966 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:43:05.966 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:43:05.966 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:43:05.966 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:43:05.966 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:43:05.966 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:43:05.969 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:43:05.970 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:43:06.913 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 106 16 9644 9628 5.67633 5.5625 0.176991 0.176022 2024-09-11T14:43:07.913 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 107 16 9735 9719 5.67642 5.6875 0.20435 0.176026 2024-09-11T14:43:08.136 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:43:08.265 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:08.261+0000 7f470bf11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:08.271 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:08.268+0000 7f470bf11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:08.914 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 108 16 9822 9806 5.67421 5.4375 0.338147 0.176074 2024-09-11T14:43:09.914 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 109 16 9911 9895 5.67318 5.5625 0.0584175 0.176088 2024-09-11T14:43:10.914 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 110 16 9977 9961 5.6591 4.125 0.11752 0.176444 2024-09-11T14:43:10.927 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:43:10.970 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:43:10.971 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:43:10.971 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:43:11.098 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:11.094+0000 7fe4ceb91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:11.103 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:11.101+0000 7fe4ceb91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:11.118 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:11.111+0000 7f0ed3cfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:11.125 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:11.124+0000 7f0ed3cfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:11.502 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:43:11.503 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:43:11.523 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":83,"stamp":"2024-09-11T14:43:10.731813+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":643957299,"num_objects":9821,"num_object_clones":0,"num_object_copies":29463,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":9898,"num_write_kb":629396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9872,"ondisk_log_size":9872,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":3780716,"kb_used_data":3780716,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":205410196,"statfs":{"total":214211493888,"available":210340040704,"internally_reserved":0,"allocated":3871453184,"data_stored":3871453184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":74317824,"num_objects":1134,"num_object_clones":0,"num_object_copies":3402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1134,"num_write_kb":72576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002879"},"pg_stats":[{"pgid":"3.f","version":"27'611","reported_seq":647,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:09.671615+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:43:09.671615+0000","last_peered":"2024-09-11T14:43:09.671615+0000","last_clean":"2024-09-11T14:43:09.671615+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:43:09.671615+0000","last_undegraded":"2024-09-11T14:43:09.671615+0000","last_fullsized":"2024-09-11T14:43:09.671615+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":611,"log_dups_size":0,"ondisk_log_size":611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40042496,"num_objects":611,"num_object_clones":0,"num_object_copies":1833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":611,"num_write_kb":39104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"27'652","reported_seq":688,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:05.405177+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:43:05.405177+0000","last_peered":"2024-09-11T14:43:05.405177+0000","last_clean":"2024-09-11T14:43:05.405177+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:43:05.405177+0000","last_undegraded":"2024-09-11T14:43:05.405177+0000","last_fullsized":"2024-09-11T14:43:05.405177+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":652,"log_dups_size":0,"ondisk_log_size":652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42729472,"num_objects":652,"num_object_clones":0,"num_object_copies":1956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":652,"num_write_kb":41728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"27'624","reported_seq":660,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:09.483149+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:43:09.483149+0000","last_peered":"2024-09-11T14:43:09.483149+0000","last_clean":"2024-09-11T14:43:09.483149+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:43:09.483149+0000","last_undegraded":"2024-09-11T14:43:09.483149+0000","last_fullsized":"2024-09-11T14:43:09.483149+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":624,"log_dups_size":0,"ondisk_log_size":624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40894464,"num_objects":624,"num_object_clones":0,"num_object_copies":1872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":624,"num_write_kb":39936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"27'613","reported_seq":649,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:08.741783+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:43:08.741783+0000","last_peered":"2024-09-11T14:43:08.741783+0000","last_clean":"2024-09-11T14:43:08.741783+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:43:08.741783+0000","last_undegraded":"2024-09-11T14:43:08.741783+0000","last_fullsized":"2024-09-11T14:43:08.741783+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":613,"log_dups_size":0,"ondisk_log_size":613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40173568,"num_objects":613,"num_object_clones":0,"num_object_copies":1839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":613,"num_write_kb":39232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"27'597","reported_seq":633,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:09.409502+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:43:09.409502+0000","last_peered":"2024-09-11T14:43:09.409502+0000","last_clean":"2024-09-11T14:43:09.409502+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:43:09.409502+0000","last_undegraded":"2024-09-11T14:43:09.409502+0000","last_fullsized":"2024-09-11T14:43:09.409502+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":597,"log_dups_size":0,"ondisk_log_size":597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39124992,"num_objects":597,"num_object_clones":0,"num_object_copies":1791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":597,"num_write_kb":38208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"27'675","reported_seq":711,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:09.473570+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:43:09.473570+0000","last_peered":"2024-09-11T14:43:09.473570+0000","last_clean":"2024-09-11T14:43:09.473570+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:43:09.473570+0000","last_undegraded":"2024-09-11T14:43:09.473570+0000","last_fullsized":"2024-09-11T14:43:09.473570+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":675,"log_dups_size":0,"ondisk_log_size":675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44236800,"num_objects":675,"num_object_clones":0,"num_object_copies":2025,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":675,"num_write_kb":43200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"27'639","reported_seq":675,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:09.401450+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:43:09.401450+0000","last_peered":"2024-09-11T14:43:09.401450+0000","last_clean":"2024-09-11T14:43:09.401450+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:43:09.401450+0000","last_undegraded":"2024-09-11T14:43:09.401450+0000","last_fullsized":"2024-09-11T14:43:09.401450+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":639,"log_dups_size":0,"ondisk_log_size":639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41877504,"num_objects":639,"num_object_clones":0,"num_object_copies":1917,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":639,"num_write_kb":40896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"27'610","reported_seq":646,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:09.055450+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:43:09.055450+0000","last_peered":"2024-09-11T14:43:09.055450+0000","last_clean":"2024-09-11T14:43:09.055450+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:43:09.055450+0000","last_undegraded":"2024-09-11T14:43:09.055450+0000","last_fullsized":"2024-09-11T14:43:09.055450+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39976960,"num_objects":610,"num_object_clones":0,"num_object_copies":1830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":610,"num_write_kb":39040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":121,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084543+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:42:13.084543+0000","last_peered":"2024-09-11T14:42:13.084543+0000","last_clean":"2024-09-11T14:42:13.084543+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:42:13.084543+0000","last_undegraded":"2024-09-11T14:42:13.084543+0000","last_fullsized":"2024-09-11T14:42:13.084543+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":50,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.184283+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:42:13.184283+0000","last_peered":"2024-09-11T14:42:13.184283+0000","last_clean":"2024-09-11T14:42:13.184283+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:42:13.184283+0000","last_undegraded":"2024-09-11T14:42:13.184283+0000","last_fullsized":"2024-09-11T14:42:13.184283+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"27'599","reported_seq":635,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:09.147733+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:43:09.147733+0000","last_peered":"2024-09-11T14:43:09.147733+0000","last_clean":"2024-09-11T14:43:09.147733+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:43:09.147733+0000","last_undegraded":"2024-09-11T14:43:09.147733+0000","last_fullsized":"2024-09-11T14:43:09.147733+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":599,"log_dups_size":0,"ondisk_log_size":599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39256064,"num_objects":599,"num_object_clones":0,"num_object_copies":1797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":599,"num_write_kb":38336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084333+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:42:13.084333+0000","last_peered":"2024-09-11T14:42:13.084333+0000","last_clean":"2024-09-11T14:42:13.084333+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:42:13.084333+0000","last_undegraded":"2024-09-11T14:42:13.084333+0000","last_fullsized":"2024-09-11T14:42:13.084333+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"27'599","reported_seq":635,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:09.363707+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:43:09.363707+0000","last_peered":"2024-09-11T14:43:09.363707+0000","last_clean":"2024-09-11T14:43:09.363707+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:43:09.363707+0000","last_undegraded":"2024-09-11T14:43:09.363707+0000","last_fullsized":"2024-09-11T14:43:09.363707+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":599,"log_dups_size":0,"ondisk_log_size":599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39256064,"num_objects":599,"num_object_clones":0,"num_object_copies":1797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":599,"num_write_kb":38336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.179633+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:42:13.179633+0000","last_peered":"2024-09-11T14:42:13.179633+0000","last_clean":"2024-09-11T14:42:13.179633+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:42:13.179633+0000","last_undegraded":"2024-09-11T14:42:13.179633+0000","last_fullsized":"2024-09-11T14:42:13.179633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"27'611","reported_seq":647,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:09.539767+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:43:09.539767+0000","last_peered":"2024-09-11T14:43:09.539767+0000","last_clean":"2024-09-11T14:43:09.539767+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:43:09.539767+0000","last_undegraded":"2024-09-11T14:43:09.539767+0000","last_fullsized":"2024-09-11T14:43:09.539767+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":611,"log_dups_size":0,"ondisk_log_size":611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40042496,"num_objects":611,"num_object_clones":0,"num_object_copies":1833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":611,"num_write_kb":39104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":51,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.082423+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:42:13.082423+0000","last_peered":"2024-09-11T14:42:13.082423+0000","last_clean":"2024-09-11T14:42:13.082423+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:42:13.082423+0000","last_undegraded":"2024-09-11T14:42:13.082423+0000","last_fullsized":"2024-09-11T14:42:13.082423+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"27'571","reported_seq":607,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:05.563146+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:43:05.563146+0000","last_peered":"2024-09-11T14:43:05.563146+0000","last_clean":"2024-09-11T14:43:05.563146+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:43:05.563146+0000","last_undegraded":"2024-09-11T14:43:05.563146+0000","last_fullsized":"2024-09-11T14:43:05.563146+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":571,"log_dups_size":0,"ondisk_log_size":571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37421056,"num_objects":571,"num_object_clones":0,"num_object_copies":1713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":571,"num_write_kb":36544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"27'615","reported_seq":651,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:09.777019+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:43:09.777019+0000","last_peered":"2024-09-11T14:43:09.777019+0000","last_clean":"2024-09-11T14:43:09.777019+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:43:09.777019+0000","last_undegraded":"2024-09-11T14:43:09.777019+0000","last_fullsized":"2024-09-11T14:43:09.777019+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":615,"log_dups_size":0,"ondisk_log_size":615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40304640,"num_objects":615,"num_object_clones":0,"num_object_copies":1845,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":615,"num_write_kb":39360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111507+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:42:13.111507+0000","last_peered":"2024-09-11T14:42:13.111507+0000","last_clean":"2024-09-11T14:42:13.111507+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:42:13.111507+0000","last_undegraded":"2024-09-11T14:42:13.111507+0000","last_fullsized":"2024-09-11T14:42:13.111507+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"27'565","reported_seq":601,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:05.437307+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:43:05.437307+0000","last_peered":"2024-09-11T14:43:05.437307+0000","last_clean":"2024-09-11T14:43:05.437307+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:43:05.437307+0000","last_undegraded":"2024-09-11T14:43:05.437307+0000","last_fullsized":"2024-09-11T14:43:05.437307+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":565,"log_dups_size":0,"ondisk_log_size":565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37027840,"num_objects":565,"num_object_clones":0,"num_object_copies":1695,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":565,"num_write_kb":36160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.086961+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:42:13.086961+0000","last_peered":"2024-09-11T14:42:13.086961+0000","last_clean":"2024-09-11T14:42:13.086961+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:42:13.086961+0000","last_undegraded":"2024-09-11T14:42:13.086961+0000","last_fullsized":"2024-09-11T14:42:13.086961+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"27'619","reported_seq":655,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:09.554263+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:43:09.554263+0000","last_peered":"2024-09-11T14:43:09.554263+0000","last_clean":"2024-09-11T14:43:09.554263+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:43:09.554263+0000","last_undegraded":"2024-09-11T14:43:09.554263+0000","last_fullsized":"2024-09-11T14:43:09.554263+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":619,"log_dups_size":0,"ondisk_log_size":619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40566784,"num_objects":619,"num_object_clones":0,"num_object_copies":1857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":619,"num_write_kb":39616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111171+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:42:13.111171+0000","last_peered":"2024-09-11T14:42:13.111171+0000","last_clean":"2024-09-11T14:42:13.111171+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:42:13.111171+0000","last_undegraded":"2024-09-11T14:42:13.111171+0000","last_fullsized":"2024-09-11T14:42:13.111171+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"27'617","reported_seq":653,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:05.662200+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:43:05.662200+0000","last_peered":"2024-09-11T14:43:05.662200+0000","last_clean":"2024-09-11T14:43:05.662200+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:43:05.662200+0000","last_undegraded":"2024-09-11T14:43:05.662200+0000","last_fullsized":"2024-09-11T14:43:05.662200+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":617,"log_dups_size":0,"ondisk_log_size":617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40435712,"num_objects":617,"num_object_clones":0,"num_object_copies":1851,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":617,"num_write_kb":39488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.181385+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:42:13.181385+0000","last_peered":"2024-09-11T14:42:13.181385+0000","last_clean":"2024-09-11T14:42:13.181385+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:42:13.181385+0000","last_undegraded":"2024-09-11T14:42:13.181385+0000","last_fullsized":"2024-09-11T14:42:13.181385+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":643366912,"num_objects":9817,"num_object_clones":0,"num_object_copies":29451,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9817,"num_write_kb":628288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":210310860800,"internally_reserved":0,"allocated":3900633088,"data_stored":3900633088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9817,"ondisk_log_size":9817,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":210310991872,"internally_reserved":0,"allocated":3900502016,"data_stored":3900502016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":52416671744,"internally_reserved":0,"allocated":1136201728,"data_stored":1136201728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673079,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":848868,"kb_used_data":848868,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51448860,"statfs":{"total":53552873472,"available":52683632640,"internally_reserved":0,"allocated":869240832,"data_stored":869240832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705788,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1101728,"kb_used_data":1101728,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51196000,"statfs":{"total":53552873472,"available":52424704000,"internally_reserved":0,"allocated":1128169472,"data_stored":1128169472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705788,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":962788,"kb_used_data":962788,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51334940,"statfs":{"total":53552873472,"available":52566978560,"internally_reserved":0,"allocated":985894912,"data_stored":985894912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705788,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":867332,"kb_used_data":867332,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51430396,"statfs":{"total":53552873472,"available":52664725504,"internally_reserved":0,"allocated":888147968,"data_stored":888147968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":52416671744,"internally_reserved":0,"allocated":1136201728,"data_stored":1136201728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":52658356224,"internally_reserved":0,"allocated":894517248,"data_stored":894517248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":52558405632,"internally_reserved":0,"allocated":994467840,"data_stored":994467840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":52677689344,"internally_reserved":0,"allocated":875184128,"data_stored":875184128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":52416540672,"internally_reserved":0,"allocated":1136332800,"data_stored":1136332800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":52658356224,"internally_reserved":0,"allocated":894517248,"data_stored":894517248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":52558405632,"internally_reserved":0,"allocated":994467840,"data_stored":994467840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":52677689344,"internally_reserved":0,"allocated":875184128,"data_stored":875184128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":52416409600,"internally_reserved":0,"allocated":1136463872,"data_stored":1136463872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:43:11.914 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 111 16 10055 10039 5.65203 4.875 0.146164 0.176748 2024-09-11T14:43:12.914 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 112 16 10132 10116 5.64453 4.8125 0.25747 0.176967 2024-09-11T14:43:13.593 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:43:13.728 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:13.725+0000 7f13aaabd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:13.733 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:13.732+0000 7f13aaabd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:13.914 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 113 16 10210 10194 5.63772 4.875 0.250877 0.17705 2024-09-11T14:43:14.914 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 114 16 10285 10269 5.62938 4.6875 0.0447214 0.177332 2024-09-11T14:43:15.914 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 115 16 10361 10345 5.62173 4.75 0.171445 0.177707 2024-09-11T14:43:16.116 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:43:16.265 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:16.261+0000 7fbaa264f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:16.271 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:16.270+0000 7fbaa264f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:16.524 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:43:16.525 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:43:16.525 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:43:16.691 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:16.684+0000 7ff8fe991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:16.697 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:16.696+0000 7ff8fe991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:16.914 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 116 16 10433 10417 5.61205 4.5 0.235122 0.177968 2024-09-11T14:43:17.059 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:43:17.060 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:43:17.081 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2024-09-11T14:43:16.733153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":675414579,"num_objects":10301,"num_object_clones":0,"num_object_copies":30903,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":10378,"num_write_kb":660116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10352,"ondisk_log_size":10352,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":3969148,"kb_used_data":3969148,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":205221764,"statfs":{"total":214211493888,"available":210147086336,"internally_reserved":0,"allocated":4064407552,"data_stored":4064407552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":78708736,"num_objects":1201,"num_object_clones":0,"num_object_copies":3603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1201,"num_write_kb":76864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002766"},"pg_stats":[{"pgid":"3.f","version":"27'630","reported_seq":666,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:14.411232+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:43:14.411232+0000","last_peered":"2024-09-11T14:43:14.411232+0000","last_clean":"2024-09-11T14:43:14.411232+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:43:14.411232+0000","last_undegraded":"2024-09-11T14:43:14.411232+0000","last_fullsized":"2024-09-11T14:43:14.411232+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":630,"log_dups_size":0,"ondisk_log_size":630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41287680,"num_objects":630,"num_object_clones":0,"num_object_copies":1890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":630,"num_write_kb":40320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"27'716","reported_seq":752,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:15.448766+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:43:15.448766+0000","last_peered":"2024-09-11T14:43:15.448766+0000","last_clean":"2024-09-11T14:43:15.448766+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:43:15.448766+0000","last_undegraded":"2024-09-11T14:43:15.448766+0000","last_fullsized":"2024-09-11T14:43:15.448766+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":716,"log_dups_size":0,"ondisk_log_size":716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46923776,"num_objects":716,"num_object_clones":0,"num_object_copies":2148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":716,"num_write_kb":45824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"27'652","reported_seq":688,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:14.725980+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:43:14.725980+0000","last_peered":"2024-09-11T14:43:14.725980+0000","last_clean":"2024-09-11T14:43:14.725980+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:43:14.725980+0000","last_undegraded":"2024-09-11T14:43:14.725980+0000","last_fullsized":"2024-09-11T14:43:14.725980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":652,"log_dups_size":0,"ondisk_log_size":652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42729472,"num_objects":652,"num_object_clones":0,"num_object_copies":1956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":652,"num_write_kb":41728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"27'634","reported_seq":670,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:13.516172+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:43:13.516172+0000","last_peered":"2024-09-11T14:43:13.516172+0000","last_clean":"2024-09-11T14:43:13.516172+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:43:13.516172+0000","last_undegraded":"2024-09-11T14:43:13.516172+0000","last_fullsized":"2024-09-11T14:43:13.516172+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":634,"log_dups_size":0,"ondisk_log_size":634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41549824,"num_objects":634,"num_object_clones":0,"num_object_copies":1902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":634,"num_write_kb":40576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"27'625","reported_seq":661,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:14.677493+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:43:14.677493+0000","last_peered":"2024-09-11T14:43:14.677493+0000","last_clean":"2024-09-11T14:43:14.677493+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:43:14.677493+0000","last_undegraded":"2024-09-11T14:43:14.677493+0000","last_fullsized":"2024-09-11T14:43:14.677493+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":625,"log_dups_size":0,"ondisk_log_size":625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40960000,"num_objects":625,"num_object_clones":0,"num_object_copies":1875,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":625,"num_write_kb":40000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"27'696","reported_seq":732,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:14.593423+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:43:14.593423+0000","last_peered":"2024-09-11T14:43:14.593423+0000","last_clean":"2024-09-11T14:43:14.593423+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:43:14.593423+0000","last_undegraded":"2024-09-11T14:43:14.593423+0000","last_fullsized":"2024-09-11T14:43:14.593423+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":696,"log_dups_size":0,"ondisk_log_size":696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45613056,"num_objects":696,"num_object_clones":0,"num_object_copies":2088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":696,"num_write_kb":44544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"27'667","reported_seq":703,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:14.243009+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:43:14.243009+0000","last_peered":"2024-09-11T14:43:14.243009+0000","last_clean":"2024-09-11T14:43:14.243009+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:43:14.243009+0000","last_undegraded":"2024-09-11T14:43:14.243009+0000","last_fullsized":"2024-09-11T14:43:14.243009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":667,"log_dups_size":0,"ondisk_log_size":667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43712512,"num_objects":667,"num_object_clones":0,"num_object_copies":2001,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":667,"num_write_kb":42688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"27'631","reported_seq":667,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:14.663825+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:43:14.663825+0000","last_peered":"2024-09-11T14:43:14.663825+0000","last_clean":"2024-09-11T14:43:14.663825+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:43:14.663825+0000","last_undegraded":"2024-09-11T14:43:14.663825+0000","last_fullsized":"2024-09-11T14:43:14.663825+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":631,"log_dups_size":0,"ondisk_log_size":631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41353216,"num_objects":631,"num_object_clones":0,"num_object_copies":1893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":631,"num_write_kb":40384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":121,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084543+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:42:13.084543+0000","last_peered":"2024-09-11T14:42:13.084543+0000","last_clean":"2024-09-11T14:42:13.084543+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:42:13.084543+0000","last_undegraded":"2024-09-11T14:42:13.084543+0000","last_fullsized":"2024-09-11T14:42:13.084543+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":50,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.184283+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:42:13.184283+0000","last_peered":"2024-09-11T14:42:13.184283+0000","last_clean":"2024-09-11T14:42:13.184283+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:42:13.184283+0000","last_undegraded":"2024-09-11T14:42:13.184283+0000","last_fullsized":"2024-09-11T14:42:13.184283+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"27'618","reported_seq":654,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:14.492423+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:43:14.492423+0000","last_peered":"2024-09-11T14:43:14.492423+0000","last_clean":"2024-09-11T14:43:14.492423+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:43:14.492423+0000","last_undegraded":"2024-09-11T14:43:14.492423+0000","last_fullsized":"2024-09-11T14:43:14.492423+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":618,"log_dups_size":0,"ondisk_log_size":618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40501248,"num_objects":618,"num_object_clones":0,"num_object_copies":1854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":618,"num_write_kb":39552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084333+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:42:13.084333+0000","last_peered":"2024-09-11T14:42:13.084333+0000","last_clean":"2024-09-11T14:42:13.084333+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:42:13.084333+0000","last_undegraded":"2024-09-11T14:42:13.084333+0000","last_fullsized":"2024-09-11T14:42:13.084333+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"27'620","reported_seq":656,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:14.171716+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:43:14.171716+0000","last_peered":"2024-09-11T14:43:14.171716+0000","last_clean":"2024-09-11T14:43:14.171716+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:43:14.171716+0000","last_undegraded":"2024-09-11T14:43:14.171716+0000","last_fullsized":"2024-09-11T14:43:14.171716+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":620,"log_dups_size":0,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40632320,"num_objects":620,"num_object_clones":0,"num_object_copies":1860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":620,"num_write_kb":39680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.179633+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:42:13.179633+0000","last_peered":"2024-09-11T14:42:13.179633+0000","last_clean":"2024-09-11T14:42:13.179633+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:42:13.179633+0000","last_undegraded":"2024-09-11T14:42:13.179633+0000","last_fullsized":"2024-09-11T14:42:13.179633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"27'640","reported_seq":676,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:14.261649+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:43:14.261649+0000","last_peered":"2024-09-11T14:43:14.261649+0000","last_clean":"2024-09-11T14:43:14.261649+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:43:14.261649+0000","last_undegraded":"2024-09-11T14:43:14.261649+0000","last_fullsized":"2024-09-11T14:43:14.261649+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":640,"log_dups_size":0,"ondisk_log_size":640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":640,"num_object_clones":0,"num_object_copies":1920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":640,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":51,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.082423+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:42:13.082423+0000","last_peered":"2024-09-11T14:42:13.082423+0000","last_clean":"2024-09-11T14:42:13.082423+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:42:13.082423+0000","last_undegraded":"2024-09-11T14:42:13.082423+0000","last_fullsized":"2024-09-11T14:42:13.082423+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"27'619","reported_seq":655,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:15.781465+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:43:15.781465+0000","last_peered":"2024-09-11T14:43:15.781465+0000","last_clean":"2024-09-11T14:43:15.781465+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:43:15.781465+0000","last_undegraded":"2024-09-11T14:43:15.781465+0000","last_fullsized":"2024-09-11T14:43:15.781465+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":619,"log_dups_size":0,"ondisk_log_size":619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40566784,"num_objects":619,"num_object_clones":0,"num_object_copies":1857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":619,"num_write_kb":39616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"27'646","reported_seq":682,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:14.350514+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:43:14.350514+0000","last_peered":"2024-09-11T14:43:14.350514+0000","last_clean":"2024-09-11T14:43:14.350514+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:43:14.350514+0000","last_undegraded":"2024-09-11T14:43:14.350514+0000","last_fullsized":"2024-09-11T14:43:14.350514+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42336256,"num_objects":646,"num_object_clones":0,"num_object_copies":1938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":646,"num_write_kb":41344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111507+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:42:13.111507+0000","last_peered":"2024-09-11T14:42:13.111507+0000","last_clean":"2024-09-11T14:42:13.111507+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:42:13.111507+0000","last_undegraded":"2024-09-11T14:42:13.111507+0000","last_fullsized":"2024-09-11T14:42:13.111507+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"27'603","reported_seq":639,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:15.766447+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:43:15.766447+0000","last_peered":"2024-09-11T14:43:15.766447+0000","last_clean":"2024-09-11T14:43:15.766447+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:43:15.766447+0000","last_undegraded":"2024-09-11T14:43:15.766447+0000","last_fullsized":"2024-09-11T14:43:15.766447+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":603,"log_dups_size":0,"ondisk_log_size":603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39518208,"num_objects":603,"num_object_clones":0,"num_object_copies":1809,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":603,"num_write_kb":38592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.086961+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:42:13.086961+0000","last_peered":"2024-09-11T14:42:13.086961+0000","last_clean":"2024-09-11T14:42:13.086961+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:42:13.086961+0000","last_undegraded":"2024-09-11T14:42:13.086961+0000","last_fullsized":"2024-09-11T14:42:13.086961+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"27'639","reported_seq":675,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:14.172139+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:43:14.172139+0000","last_peered":"2024-09-11T14:43:14.172139+0000","last_clean":"2024-09-11T14:43:14.172139+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:43:14.172139+0000","last_undegraded":"2024-09-11T14:43:14.172139+0000","last_fullsized":"2024-09-11T14:43:14.172139+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":639,"log_dups_size":0,"ondisk_log_size":639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41877504,"num_objects":639,"num_object_clones":0,"num_object_copies":1917,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":639,"num_write_kb":40896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111171+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:42:13.111171+0000","last_peered":"2024-09-11T14:42:13.111171+0000","last_clean":"2024-09-11T14:42:13.111171+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:42:13.111171+0000","last_undegraded":"2024-09-11T14:42:13.111171+0000","last_fullsized":"2024-09-11T14:42:13.111171+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"27'661","reported_seq":697,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:14.744371+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:43:14.744371+0000","last_peered":"2024-09-11T14:43:14.744371+0000","last_clean":"2024-09-11T14:43:14.744371+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:43:14.744371+0000","last_undegraded":"2024-09-11T14:43:14.744371+0000","last_fullsized":"2024-09-11T14:43:14.744371+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43319296,"num_objects":661,"num_object_clones":0,"num_object_copies":1983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":661,"num_write_kb":42304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.181385+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:42:13.181385+0000","last_peered":"2024-09-11T14:42:13.181385+0000","last_clean":"2024-09-11T14:42:13.181385+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:42:13.181385+0000","last_undegraded":"2024-09-11T14:42:13.181385+0000","last_fullsized":"2024-09-11T14:42:13.181385+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":674824192,"num_objects":10297,"num_object_clones":0,"num_object_copies":30891,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10297,"num_write_kb":659008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":210120523776,"internally_reserved":0,"allocated":4090970112,"data_stored":4090970112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10297,"ondisk_log_size":10297,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":210120654848,"internally_reserved":0,"allocated":4090839040,"data_stored":4090839040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":52372717568,"internally_reserved":0,"allocated":1180155904,"data_stored":1180155904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673089,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":921120,"kb_used_data":921120,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51376608,"statfs":{"total":53552873472,"available":52609646592,"internally_reserved":0,"allocated":943226880,"data_stored":943226880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705793,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1145024,"kb_used_data":1145024,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51152704,"statfs":{"total":53552873472,"available":52380368896,"internally_reserved":0,"allocated":1172504576,"data_stored":1172504576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705793,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1002752,"kb_used_data":1002752,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51294976,"statfs":{"total":53552873472,"available":52526055424,"internally_reserved":0,"allocated":1026818048,"data_stored":1026818048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705793,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":900252,"kb_used_data":900252,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51397476,"statfs":{"total":53552873472,"available":52631015424,"internally_reserved":0,"allocated":921858048,"data_stored":921858048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":52372717568,"internally_reserved":0,"allocated":1180155904,"data_stored":1180155904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":52626264064,"internally_reserved":0,"allocated":926609408,"data_stored":926609408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":52519337984,"internally_reserved":0,"allocated":1033535488,"data_stored":1033535488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":52602597376,"internally_reserved":0,"allocated":950276096,"data_stored":950276096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":52372455424,"internally_reserved":0,"allocated":1180418048,"data_stored":1180418048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":52626264064,"internally_reserved":0,"allocated":926609408,"data_stored":926609408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":52519337984,"internally_reserved":0,"allocated":1033535488,"data_stored":1033535488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":52602466304,"internally_reserved":0,"allocated":950407168,"data_stored":950407168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":52372455424,"internally_reserved":0,"allocated":1180418048,"data_stored":1180418048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:43:17.915 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 117 16 10518 10502 5.60949 5.3125 0.516194 0.178121 2024-09-11T14:43:18.838 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:43:18.915 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 118 16 10601 10585 5.60591 5.1875 0.0554612 0.178148 2024-09-11T14:43:18.995 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:18.991+0000 7f7ded311640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:18.999 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:18.998+0000 7f7ded311640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:19.915 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 119 16 10680 10664 5.60029 4.9375 0.152913 0.178329 2024-09-11T14:43:20.915 INFO:tasks.radosbench.radosbench.0.smithi181.stdout:2024-09-11T14:43:20.914846+0000 min lat: 0.0249053 max lat: 1.01958 avg lat: 0.178496 2024-09-11T14:43:20.915 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T14:43:20.915 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 120 16 10763 10747 5.59684 5.1875 0.141951 0.178496 2024-09-11T14:43:21.583 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:43:21.733 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:21.728+0000 7f1bddd91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:21.740 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:21.739+0000 7f1bddd91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:21.915 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 121 16 10849 10833 5.59501 5.375 0.289751 0.178586 2024-09-11T14:43:22.083 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:43:22.083 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:43:22.083 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-11T14:43:22.084 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-11T14:43:22.242 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:22.231+0000 7fc1b0f51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:22.249 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:22.248+0000 7fc1b0f51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:22.475 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:43:22.476 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:43:22.476 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:43:22.476 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:43:22.476 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:43:22.476 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:43:22.476 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:43:22.476 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:43:22.476 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:43:22.476 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:43:22.476 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:43:22.476 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:43:22.480 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:43:22.481 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:43:22.481 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-11T14:43:22.668 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:22.656+0000 7fd2c04a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:22.675 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:22.674+0000 7fd2c04a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:22.892 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:43:22.892 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:43:22.892 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:43:22.892 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:43:22.892 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:43:22.892 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:43:22.892 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:43:22.892 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:43:22.893 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:43:22.893 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:43:22.893 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:43:22.893 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:43:22.897 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:43:22.898 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:43:22.898 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-11T14:43:22.915 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 122 16 10937 10921 5.59422 5.5 0.171393 0.178603 2024-09-11T14:43:23.041 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:23.033+0000 7f2bc4911640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:23.045 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:23.044+0000 7f2bc4911640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:23.135 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:43:23.135 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:43:23.135 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:43:23.135 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:43:23.135 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:43:23.136 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:43:23.136 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:43:23.136 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:43:23.136 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:43:23.136 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:43:23.136 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:43:23.136 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:43:23.139 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:43:23.139 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:43:23.139 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-11T14:43:23.318 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:23.308+0000 7fb8ebf91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:23.326 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:23.325+0000 7fb8ebf91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:23.455 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:43:23.455 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:43:23.455 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:43:23.455 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:43:23.455 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:43:23.456 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:43:23.456 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:43:23.456 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:43:23.456 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:43:23.456 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:43:23.456 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:43:23.456 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:43:23.459 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:43:23.459 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:43:23.915 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 123 16 11022 11006 5.59193 5.3125 0.156549 0.17865 2024-09-11T14:43:24.383 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:43:24.525 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:24.521+0000 7f03d027d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:24.530 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:24.528+0000 7f03d027d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:24.915 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 124 16 11099 11083 5.58564 4.8125 0.265356 0.178744 2024-09-11T14:43:25.915 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 125 16 11182 11166 5.58245 5.1875 0.116091 0.178963 2024-09-11T14:43:26.915 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 126 16 11262 11246 5.57782 5 0.0487443 0.179068 2024-09-11T14:43:27.145 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:43:27.298 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:27.294+0000 7f26428d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:27.304 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:27.303+0000 7f26428d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:27.916 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 127 16 11347 11331 5.57573 5.3125 0.154075 0.17916 2024-09-11T14:43:28.460 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:43:28.461 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:43:28.461 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-11T14:43:28.461 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-11T14:43:28.630 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:28.622+0000 7f32ee191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:28.636 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:28.635+0000 7f32ee191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:28.659 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:43:28.659 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:43:28.659 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:43:28.659 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:43:28.659 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:43:28.659 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:43:28.659 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:43:28.659 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:43:28.659 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:43:28.660 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:43:28.660 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:43:28.660 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:43:28.662 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:43:28.662 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:43:28.663 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-11T14:43:28.808 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:28.796+0000 7efe8849c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:28.816 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:28.814+0000 7efe8849c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:28.916 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 128 16 11423 11407 5.56928 4.75 0.0962504 0.179348 2024-09-11T14:43:28.983 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:43:28.983 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:43:28.983 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:43:28.983 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:43:28.983 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:43:28.983 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:43:28.983 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:43:28.983 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:43:28.983 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:43:28.984 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:43:28.984 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:43:28.984 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:43:28.986 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:43:28.986 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:43:28.987 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-11T14:43:29.126 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:29.114+0000 7fa3d50c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:29.131 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:29.130+0000 7fa3d50c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:29.228 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:43:29.228 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:43:29.228 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:43:29.228 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:43:29.228 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:43:29.228 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:43:29.228 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:43:29.228 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:43:29.228 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:43:29.228 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:43:29.228 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:43:29.229 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:43:29.231 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:43:29.232 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:43:29.232 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-11T14:43:29.363 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:29.355+0000 7f84db911640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:29.368 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:29.367+0000 7f84db911640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:29.534 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:43:29.534 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:43:29.534 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:43:29.534 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:43:29.534 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:43:29.534 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:43:29.534 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:43:29.534 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:43:29.534 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:43:29.535 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:43:29.535 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:43:29.535 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:43:29.538 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:43:29.538 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:43:29.793 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:43:29.916 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 129 16 11502 11486 5.56437 4.9375 0.246976 0.179498 2024-09-11T14:43:29.930 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:29.927+0000 7fae86591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:29.935 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:29.934+0000 7fae86591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:30.916 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 130 16 11584 11568 5.56099 5.125 0.27402 0.179668 2024-09-11T14:43:31.916 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 131 16 11666 11650 5.55766 5.125 0.199515 0.179746 2024-09-11T14:43:32.430 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:43:32.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:32.569+0000 7feccd991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:32.577 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:32.576+0000 7feccd991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:32.916 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 132 16 11739 11723 5.55012 4.5625 0.197143 0.179924 2024-09-11T14:43:33.916 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 133 16 11825 11809 5.5488 5.375 0.221107 0.180093 2024-09-11T14:43:34.539 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:43:34.539 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:43:34.539 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-11T14:43:34.539 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-11T14:43:34.692 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:34.680+0000 7f8b9eed1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:34.698 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:34.697+0000 7f8b9eed1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:34.726 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:43:34.727 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:43:34.727 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:43:34.727 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:43:34.727 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:43:34.727 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:43:34.727 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:43:34.727 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:43:34.727 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:43:34.727 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:43:34.727 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:43:34.727 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:43:34.731 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:43:34.732 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:43:34.732 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-11T14:43:34.897 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:34.889+0000 7fd924d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:34.902 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:34.901+0000 7fd924d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:34.916 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 134 16 11904 11888 5.54423 4.9375 0.163684 0.18021 2024-09-11T14:43:35.079 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:43:35.079 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:43:35.079 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:43:35.079 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:43:35.079 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:43:35.080 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:43:35.080 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:43:35.080 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:43:35.080 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:43:35.080 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:43:35.080 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:43:35.080 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:43:35.083 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:43:35.084 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:43:35.084 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-11T14:43:35.183 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:43:35.269 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:35.255+0000 7f4fe2991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:35.283 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:35.280+0000 7f4fe2991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:35.366 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:35.361+0000 7f1935591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:35.374 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:35.373+0000 7f1935591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:35.392 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:43:35.392 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:43:35.393 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:43:35.393 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:43:35.393 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:43:35.393 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:43:35.393 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:43:35.393 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:43:35.393 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:43:35.393 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:43:35.393 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:43:35.393 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:43:35.397 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:43:35.397 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:43:35.397 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-11T14:43:35.559 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:35.551+0000 7fc907a6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:35.566 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:35.565+0000 7fc907a6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:35.665 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:43:35.666 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:43:35.666 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:43:35.666 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:43:35.666 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:43:35.666 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:43:35.666 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:43:35.666 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:43:35.666 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:43:35.666 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:43:35.666 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:43:35.666 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:43:35.670 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:43:35.670 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:43:35.916 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 135 16 11980 11964 5.53834 4.75 0.224096 0.18033 2024-09-11T14:43:36.916 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 136 16 12054 12038 5.53163 4.625 0.247298 0.180603 2024-09-11T14:43:37.916 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 137 16 12130 12114 5.52592 4.75 0.170167 0.180826 2024-09-11T14:43:38.073 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:43:38.250 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:38.246+0000 7f555dd91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:38.257 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:38.256+0000 7f555dd91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:38.917 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 138 16 12210 12194 5.5221 5 0.0981227 0.180895 2024-09-11T14:43:39.917 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 139 16 12285 12269 5.5161 4.6875 0.166235 0.181054 2024-09-11T14:43:40.671 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:43:40.672 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:43:40.672 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-11T14:43:40.672 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-11T14:43:40.692 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:43:40.829 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:40.816+0000 7f856a991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:40.845 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:40.838+0000 7f856972f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:40.898 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:40.894+0000 7f0b70991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:40.904 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:40.902+0000 7f0b70991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:40.917 INFO:tasks.radosbench.radosbench.0.smithi181.stdout:2024-09-11T14:43:40.916741+0000 min lat: 0.0249053 max lat: 1.01958 avg lat: 0.181244 2024-09-11T14:43:40.917 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-11T14:43:40.917 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 140 16 12368 12352 5.51375 5.1875 0.0895946 0.181244 2024-09-11T14:43:40.918 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:43:40.919 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:43:40.919 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:43:40.919 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:43:40.919 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:43:40.919 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:43:40.919 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:43:40.919 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:43:40.919 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:43:40.919 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:43:40.919 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:43:40.919 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:43:40.924 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:43:40.924 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:43:40.925 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-11T14:43:41.105 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:41.096+0000 7f2cb7493640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:41.110 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:41.109+0000 7f2cb7493640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:41.262 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:43:41.262 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:43:41.262 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:43:41.262 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:43:41.262 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:43:41.262 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:43:41.262 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:43:41.262 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:43:41.262 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:43:41.263 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:43:41.263 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:43:41.263 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:43:41.265 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:43:41.265 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:43:41.265 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-11T14:43:41.411 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:41.403+0000 7fde90b51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:41.418 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:41.416+0000 7fde90b51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:41.438 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:43:41.438 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:43:41.438 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:43:41.438 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:43:41.439 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:43:41.439 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:43:41.439 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:43:41.439 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:43:41.439 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:43:41.439 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:43:41.439 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:43:41.439 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:43:41.442 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:43:41.442 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:43:41.442 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-11T14:43:41.609 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:41.601+0000 7f47b5991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:41.613 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:41.612+0000 7f47b5991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:41.737 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:43:41.737 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:43:41.738 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:43:41.738 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:43:41.738 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:43:41.738 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:43:41.738 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:43:41.738 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:43:41.738 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:43:41.738 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:43:41.738 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:43:41.738 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:43:41.740 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:43:41.741 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:43:41.920 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 141 16 12429 12413 5.50168 3.8125 0.244039 0.181484 2024-09-11T14:43:42.917 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 142 16 12508 12492 5.4977 4.9375 0.093516 0.181732 2024-09-11T14:43:43.432 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:43:43.583 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:43.579+0000 7fb1a6f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:43.588 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:43.587+0000 7fb1a6f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:43.917 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 143 16 12584 12568 5.49247 4.75 0.271452 0.181854 2024-09-11T14:43:44.917 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 144 16 12659 12643 5.48688 4.6875 0.414438 0.18203 2024-09-11T14:43:45.917 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 145 16 12734 12718 5.48136 4.6875 0.395571 0.182187 2024-09-11T14:43:46.148 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:43:46.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:46.324+0000 7f1652391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:46.332 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:46.331+0000 7f1652391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:46.742 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:43:46.742 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:43:46.742 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:43:46.898 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:46.891+0000 7f7bf3d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:46.903 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:46.902+0000 7f7bf3d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:46.917 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 146 16 12802 12786 5.47292 4.25 0.472306 0.182499 2024-09-11T14:43:47.269 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:43:47.269 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:43:47.296 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-09-11T14:43:46.740241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":830865971,"num_objects":12673,"num_object_clones":0,"num_object_copies":38019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":12750,"num_write_kb":811924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12724,"ondisk_log_size":12724,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":4892064,"kb_used_data":4892064,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":204298848,"statfs":{"total":214211493888,"available":209202020352,"internally_reserved":0,"allocated":5009473536,"data_stored":5009473536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":76087296,"num_objects":1161,"num_object_clones":0,"num_object_copies":3483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1161,"num_write_kb":74304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002854"},"pg_stats":[{"pgid":"3.f","version":"27'791","reported_seq":827,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:44.155013+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:43:44.155013+0000","last_peered":"2024-09-11T14:43:44.155013+0000","last_clean":"2024-09-11T14:43:44.155013+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:43:44.155013+0000","last_undegraded":"2024-09-11T14:43:44.155013+0000","last_fullsized":"2024-09-11T14:43:44.155013+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":791,"log_dups_size":0,"ondisk_log_size":791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51838976,"num_objects":791,"num_object_clones":0,"num_object_copies":2373,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":791,"num_write_kb":50624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"27'865","reported_seq":901,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:45.608946+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:43:45.608946+0000","last_peered":"2024-09-11T14:43:45.608946+0000","last_clean":"2024-09-11T14:43:45.608946+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:43:45.608946+0000","last_undegraded":"2024-09-11T14:43:45.608946+0000","last_fullsized":"2024-09-11T14:43:45.608946+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":865,"log_dups_size":0,"ondisk_log_size":865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56688640,"num_objects":865,"num_object_clones":0,"num_object_copies":2595,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":865,"num_write_kb":55360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"27'803","reported_seq":839,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:44.597310+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:43:44.597310+0000","last_peered":"2024-09-11T14:43:44.597310+0000","last_clean":"2024-09-11T14:43:44.597310+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:43:44.597310+0000","last_undegraded":"2024-09-11T14:43:44.597310+0000","last_fullsized":"2024-09-11T14:43:44.597310+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":803,"log_dups_size":0,"ondisk_log_size":803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52625408,"num_objects":803,"num_object_clones":0,"num_object_copies":2409,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":803,"num_write_kb":51392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"27'758","reported_seq":794,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:44.389803+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:43:44.389803+0000","last_peered":"2024-09-11T14:43:44.389803+0000","last_clean":"2024-09-11T14:43:44.389803+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:43:44.389803+0000","last_undegraded":"2024-09-11T14:43:44.389803+0000","last_fullsized":"2024-09-11T14:43:44.389803+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":758,"log_dups_size":0,"ondisk_log_size":758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49676288,"num_objects":758,"num_object_clones":0,"num_object_copies":2274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":758,"num_write_kb":48512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"27'768","reported_seq":804,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:44.788569+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:43:44.788569+0000","last_peered":"2024-09-11T14:43:44.788569+0000","last_clean":"2024-09-11T14:43:44.788569+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:43:44.788569+0000","last_undegraded":"2024-09-11T14:43:44.788569+0000","last_fullsized":"2024-09-11T14:43:44.788569+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":768,"log_dups_size":0,"ondisk_log_size":768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":768,"num_object_clones":0,"num_object_copies":2304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":768,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"27'858","reported_seq":894,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:44.260613+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:43:44.260613+0000","last_peered":"2024-09-11T14:43:44.260613+0000","last_clean":"2024-09-11T14:43:44.260613+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:43:44.260613+0000","last_undegraded":"2024-09-11T14:43:44.260613+0000","last_fullsized":"2024-09-11T14:43:44.260613+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":858,"log_dups_size":0,"ondisk_log_size":858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56229888,"num_objects":858,"num_object_clones":0,"num_object_copies":2574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":858,"num_write_kb":54912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"27'810","reported_seq":846,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:44.451751+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:43:44.451751+0000","last_peered":"2024-09-11T14:43:44.451751+0000","last_clean":"2024-09-11T14:43:44.451751+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:43:44.451751+0000","last_undegraded":"2024-09-11T14:43:44.451751+0000","last_fullsized":"2024-09-11T14:43:44.451751+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":810,"log_dups_size":0,"ondisk_log_size":810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53084160,"num_objects":810,"num_object_clones":0,"num_object_copies":2430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":810,"num_write_kb":51840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"27'769","reported_seq":805,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:44.567516+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:43:44.567516+0000","last_peered":"2024-09-11T14:43:44.567516+0000","last_clean":"2024-09-11T14:43:44.567516+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:43:44.567516+0000","last_undegraded":"2024-09-11T14:43:44.567516+0000","last_fullsized":"2024-09-11T14:43:44.567516+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":769,"log_dups_size":0,"ondisk_log_size":769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50397184,"num_objects":769,"num_object_clones":0,"num_object_copies":2307,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":769,"num_write_kb":49216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":121,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084543+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:42:13.084543+0000","last_peered":"2024-09-11T14:42:13.084543+0000","last_clean":"2024-09-11T14:42:13.084543+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:42:13.084543+0000","last_undegraded":"2024-09-11T14:42:13.084543+0000","last_fullsized":"2024-09-11T14:42:13.084543+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":50,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.184283+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:42:13.184283+0000","last_peered":"2024-09-11T14:42:13.184283+0000","last_clean":"2024-09-11T14:42:13.184283+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:42:13.184283+0000","last_undegraded":"2024-09-11T14:42:13.184283+0000","last_fullsized":"2024-09-11T14:42:13.184283+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"27'774","reported_seq":810,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:44.128971+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:43:44.128971+0000","last_peered":"2024-09-11T14:43:44.128971+0000","last_clean":"2024-09-11T14:43:44.128971+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:43:44.128971+0000","last_undegraded":"2024-09-11T14:43:44.128971+0000","last_fullsized":"2024-09-11T14:43:44.128971+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":774,"log_dups_size":0,"ondisk_log_size":774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50724864,"num_objects":774,"num_object_clones":0,"num_object_copies":2322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":774,"num_write_kb":49536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084333+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:42:13.084333+0000","last_peered":"2024-09-11T14:42:13.084333+0000","last_clean":"2024-09-11T14:42:13.084333+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:42:13.084333+0000","last_undegraded":"2024-09-11T14:42:13.084333+0000","last_fullsized":"2024-09-11T14:42:13.084333+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"27'792","reported_seq":828,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:44.635203+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:43:44.635203+0000","last_peered":"2024-09-11T14:43:44.635203+0000","last_clean":"2024-09-11T14:43:44.635203+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:43:44.635203+0000","last_undegraded":"2024-09-11T14:43:44.635203+0000","last_fullsized":"2024-09-11T14:43:44.635203+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":792,"log_dups_size":0,"ondisk_log_size":792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51904512,"num_objects":792,"num_object_clones":0,"num_object_copies":2376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":792,"num_write_kb":50688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.179633+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:42:13.179633+0000","last_peered":"2024-09-11T14:42:13.179633+0000","last_clean":"2024-09-11T14:42:13.179633+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:42:13.179633+0000","last_undegraded":"2024-09-11T14:42:13.179633+0000","last_fullsized":"2024-09-11T14:42:13.179633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"27'779","reported_seq":815,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:44.323932+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:43:44.323932+0000","last_peered":"2024-09-11T14:43:44.323932+0000","last_clean":"2024-09-11T14:43:44.323932+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:43:44.323932+0000","last_undegraded":"2024-09-11T14:43:44.323932+0000","last_fullsized":"2024-09-11T14:43:44.323932+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":779,"log_dups_size":0,"ondisk_log_size":779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51052544,"num_objects":779,"num_object_clones":0,"num_object_copies":2337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":779,"num_write_kb":49856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":51,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.082423+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:42:13.082423+0000","last_peered":"2024-09-11T14:42:13.082423+0000","last_clean":"2024-09-11T14:42:13.082423+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:42:13.082423+0000","last_undegraded":"2024-09-11T14:42:13.082423+0000","last_fullsized":"2024-09-11T14:42:13.082423+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"27'776","reported_seq":812,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:45.288906+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:43:45.288906+0000","last_peered":"2024-09-11T14:43:45.288906+0000","last_clean":"2024-09-11T14:43:45.288906+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:43:45.288906+0000","last_undegraded":"2024-09-11T14:43:45.288906+0000","last_fullsized":"2024-09-11T14:43:45.288906+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":776,"log_dups_size":0,"ondisk_log_size":776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50855936,"num_objects":776,"num_object_clones":0,"num_object_copies":2328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":776,"num_write_kb":49664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"27'803","reported_seq":839,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:44.627091+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:43:44.627091+0000","last_peered":"2024-09-11T14:43:44.627091+0000","last_clean":"2024-09-11T14:43:44.627091+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:43:44.627091+0000","last_undegraded":"2024-09-11T14:43:44.627091+0000","last_fullsized":"2024-09-11T14:43:44.627091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":803,"log_dups_size":0,"ondisk_log_size":803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52625408,"num_objects":803,"num_object_clones":0,"num_object_copies":2409,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":803,"num_write_kb":51392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111507+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:42:13.111507+0000","last_peered":"2024-09-11T14:42:13.111507+0000","last_clean":"2024-09-11T14:42:13.111507+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:42:13.111507+0000","last_undegraded":"2024-09-11T14:42:13.111507+0000","last_fullsized":"2024-09-11T14:42:13.111507+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"27'735","reported_seq":771,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:44.491245+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:43:44.491245+0000","last_peered":"2024-09-11T14:43:44.491245+0000","last_clean":"2024-09-11T14:43:44.491245+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:43:44.491245+0000","last_undegraded":"2024-09-11T14:43:44.491245+0000","last_fullsized":"2024-09-11T14:43:44.491245+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":735,"log_dups_size":0,"ondisk_log_size":735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48168960,"num_objects":735,"num_object_clones":0,"num_object_copies":2205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":735,"num_write_kb":47040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.086961+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:42:13.086961+0000","last_peered":"2024-09-11T14:42:13.086961+0000","last_clean":"2024-09-11T14:42:13.086961+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:42:13.086961+0000","last_undegraded":"2024-09-11T14:42:13.086961+0000","last_fullsized":"2024-09-11T14:42:13.086961+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"27'795","reported_seq":831,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:44.708000+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:43:44.708000+0000","last_peered":"2024-09-11T14:43:44.708000+0000","last_clean":"2024-09-11T14:43:44.708000+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:43:44.708000+0000","last_undegraded":"2024-09-11T14:43:44.708000+0000","last_fullsized":"2024-09-11T14:43:44.708000+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":795,"log_dups_size":0,"ondisk_log_size":795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52101120,"num_objects":795,"num_object_clones":0,"num_object_copies":2385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":795,"num_write_kb":50880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111171+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:42:13.111171+0000","last_peered":"2024-09-11T14:42:13.111171+0000","last_clean":"2024-09-11T14:42:13.111171+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:42:13.111171+0000","last_undegraded":"2024-09-11T14:42:13.111171+0000","last_fullsized":"2024-09-11T14:42:13.111171+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"27'793","reported_seq":829,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:45.359955+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:43:45.359955+0000","last_peered":"2024-09-11T14:43:45.359955+0000","last_clean":"2024-09-11T14:43:45.359955+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:43:45.359955+0000","last_undegraded":"2024-09-11T14:43:45.359955+0000","last_fullsized":"2024-09-11T14:43:45.359955+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":793,"log_dups_size":0,"ondisk_log_size":793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51970048,"num_objects":793,"num_object_clones":0,"num_object_copies":2379,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":793,"num_write_kb":50752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.181385+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:42:13.181385+0000","last_peered":"2024-09-11T14:42:13.181385+0000","last_clean":"2024-09-11T14:42:13.181385+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:42:13.181385+0000","last_undegraded":"2024-09-11T14:42:13.181385+0000","last_fullsized":"2024-09-11T14:42:13.181385+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":830275584,"num_objects":12669,"num_object_clones":0,"num_object_copies":38007,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12669,"num_write_kb":810816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":209177440256,"internally_reserved":0,"allocated":5034053632,"data_stored":5034053632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12669,"ondisk_log_size":12669,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":209177702400,"internally_reserved":0,"allocated":5033791488,"data_stored":5033791488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":52099276800,"internally_reserved":0,"allocated":1453596672,"data_stored":1453596672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673119,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1126724,"kb_used_data":1126724,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51171004,"statfs":{"total":53552873472,"available":52399108096,"internally_reserved":0,"allocated":1153765376,"data_stored":1153765376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705823,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1412808,"kb_used_data":1412808,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":50884920,"statfs":{"total":53552873472,"available":52106158080,"internally_reserved":0,"allocated":1446715392,"data_stored":1446715392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705823,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1238008,"kb_used_data":1238008,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51059720,"statfs":{"total":53552873472,"available":52285153280,"internally_reserved":0,"allocated":1267720192,"data_stored":1267720192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705823,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1114524,"kb_used_data":1114524,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51183204,"statfs":{"total":53552873472,"available":52411600896,"internally_reserved":0,"allocated":1141272576,"data_stored":1141272576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":52099276800,"internally_reserved":0,"allocated":1453596672,"data_stored":1453596672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":52406050816,"internally_reserved":0,"allocated":1146822656,"data_stored":1146822656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":52279087104,"internally_reserved":0,"allocated":1273786368,"data_stored":1273786368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":52393287680,"internally_reserved":0,"allocated":1159585792,"data_stored":1159585792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":52099276800,"internally_reserved":0,"allocated":1453596672,"data_stored":1453596672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":52405919744,"internally_reserved":0,"allocated":1146953728,"data_stored":1146953728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":52278956032,"internally_reserved":0,"allocated":1273917440,"data_stored":1273917440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":52393287680,"internally_reserved":0,"allocated":1159585792,"data_stored":1159585792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":52099276800,"internally_reserved":0,"allocated":1453596672,"data_stored":1453596672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:43:47.917 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 147 16 12875 12859 5.46673 4.5625 0.0694363 0.182733 2024-09-11T14:43:48.702 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:43:48.850 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:48.846+0000 7f8675efd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:48.855 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:48.854+0000 7f8675efd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:48.918 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 148 16 12945 12929 5.45935 4.375 0.109038 0.182994 2024-09-11T14:43:49.918 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 149 16 13030 13014 5.45836 5.3125 0.157261 0.183072 2024-09-11T14:43:50.918 INFO:tasks.radosbench.radosbench.0.smithi181.stdout: 150 14 13103 13089 5.45322 4.6875 0.0735835 0.183197 2024-09-11T14:43:51.424 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:43:51.530 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:51.527+0000 7fd2eb711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:51.534 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:51.533+0000 7fd2eb711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:51.918 INFO:tasks.radosbench.radosbench.0.smithi181.stdout:Total time run: 150.147 2024-09-11T14:43:51.918 INFO:tasks.radosbench.radosbench.0.smithi181.stdout:Total writes made: 13103 2024-09-11T14:43:51.918 INFO:tasks.radosbench.radosbench.0.smithi181.stdout:Write size: 65536 2024-09-11T14:43:51.918 INFO:tasks.radosbench.radosbench.0.smithi181.stdout:Object size: 65536 2024-09-11T14:43:51.918 INFO:tasks.radosbench.radosbench.0.smithi181.stdout:Bandwidth (MB/sec): 5.45424 2024-09-11T14:43:51.918 INFO:tasks.radosbench.radosbench.0.smithi181.stdout:Stddev Bandwidth: 0.700792 2024-09-11T14:43:51.918 INFO:tasks.radosbench.radosbench.0.smithi181.stdout:Max bandwidth (MB/sec): 7.0625 2024-09-11T14:43:51.919 INFO:tasks.radosbench.radosbench.0.smithi181.stdout:Min bandwidth (MB/sec): 3.8125 2024-09-11T14:43:51.919 INFO:tasks.radosbench.radosbench.0.smithi181.stdout:Average IOPS: 87 2024-09-11T14:43:51.919 INFO:tasks.radosbench.radosbench.0.smithi181.stdout:Stddev IOPS: 11.2127 2024-09-11T14:43:51.919 INFO:tasks.radosbench.radosbench.0.smithi181.stdout:Max IOPS: 113 2024-09-11T14:43:51.919 INFO:tasks.radosbench.radosbench.0.smithi181.stdout:Min IOPS: 61 2024-09-11T14:43:51.919 INFO:tasks.radosbench.radosbench.0.smithi181.stdout:Average Latency(s): 0.183249 2024-09-11T14:43:51.919 INFO:tasks.radosbench.radosbench.0.smithi181.stdout:Stddev Latency(s): 0.114919 2024-09-11T14:43:51.919 INFO:tasks.radosbench.radosbench.0.smithi181.stdout:Max latency(s): 1.01958 2024-09-11T14:43:51.919 INFO:tasks.radosbench.radosbench.0.smithi181.stdout:Min latency(s): 0.0249053 2024-09-11T14:43:51.958 INFO:tasks.radosbench.radosbench.0.smithi181.stdout:Cleaning up (deleting benchmark objects) 2024-09-11T14:43:52.299 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:43:52.299 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:43:52.300 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:43:52.443 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:52.436+0000 7fc0bd0a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:52.448 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:52.447+0000 7fc0bd0a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:52.773 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:43:52.774 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:43:52.791 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2024-09-11T14:43:52.741779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":855114291,"num_objects":13043,"num_object_clones":0,"num_object_copies":39129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":13120,"num_write_kb":835604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13094,"ondisk_log_size":13094,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":5031896,"kb_used_data":5031896,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":204159016,"statfs":{"total":214211493888,"available":209058832384,"internally_reserved":0,"allocated":5152661504,"data_stored":5152661504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":54657024,"num_objects":834,"num_object_clones":0,"num_object_copies":2502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":834,"num_write_kb":53376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003003"},"pg_stats":[{"pgid":"3.f","version":"27'806","reported_seq":842,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:49.285512+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:43:49.285512+0000","last_peered":"2024-09-11T14:43:49.285512+0000","last_clean":"2024-09-11T14:43:49.285512+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:43:49.285512+0000","last_undegraded":"2024-09-11T14:43:49.285512+0000","last_fullsized":"2024-09-11T14:43:49.285512+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":806,"log_dups_size":0,"ondisk_log_size":806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52822016,"num_objects":806,"num_object_clones":0,"num_object_copies":2418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":806,"num_write_kb":51584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"27'890","reported_seq":926,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:50.318600+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:43:50.318600+0000","last_peered":"2024-09-11T14:43:50.318600+0000","last_clean":"2024-09-11T14:43:50.318600+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:43:50.318600+0000","last_undegraded":"2024-09-11T14:43:50.318600+0000","last_fullsized":"2024-09-11T14:43:50.318600+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":890,"log_dups_size":0,"ondisk_log_size":890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58327040,"num_objects":890,"num_object_clones":0,"num_object_copies":2670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":890,"num_write_kb":56960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"27'827","reported_seq":863,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:49.698299+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:43:49.698299+0000","last_peered":"2024-09-11T14:43:49.698299+0000","last_clean":"2024-09-11T14:43:49.698299+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:43:49.698299+0000","last_undegraded":"2024-09-11T14:43:49.698299+0000","last_fullsized":"2024-09-11T14:43:49.698299+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":827,"log_dups_size":0,"ondisk_log_size":827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54198272,"num_objects":827,"num_object_clones":0,"num_object_copies":2481,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":827,"num_write_kb":52928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"27'782","reported_seq":818,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:49.127266+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:43:49.127266+0000","last_peered":"2024-09-11T14:43:49.127266+0000","last_clean":"2024-09-11T14:43:49.127266+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:43:49.127266+0000","last_undegraded":"2024-09-11T14:43:49.127266+0000","last_fullsized":"2024-09-11T14:43:49.127266+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":782,"log_dups_size":0,"ondisk_log_size":782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51249152,"num_objects":782,"num_object_clones":0,"num_object_copies":2346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":782,"num_write_kb":50048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"27'795","reported_seq":831,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:49.330887+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:43:49.330887+0000","last_peered":"2024-09-11T14:43:49.330887+0000","last_clean":"2024-09-11T14:43:49.330887+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:43:49.330887+0000","last_undegraded":"2024-09-11T14:43:49.330887+0000","last_fullsized":"2024-09-11T14:43:49.330887+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":795,"log_dups_size":0,"ondisk_log_size":795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52101120,"num_objects":795,"num_object_clones":0,"num_object_copies":2385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":795,"num_write_kb":50880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"27'882","reported_seq":918,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:49.683694+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:43:49.683694+0000","last_peered":"2024-09-11T14:43:49.683694+0000","last_clean":"2024-09-11T14:43:49.683694+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:43:49.683694+0000","last_undegraded":"2024-09-11T14:43:49.683694+0000","last_fullsized":"2024-09-11T14:43:49.683694+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":882,"log_dups_size":0,"ondisk_log_size":882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57802752,"num_objects":882,"num_object_clones":0,"num_object_copies":2646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":882,"num_write_kb":56448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"27'831","reported_seq":867,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:49.504288+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:43:49.504288+0000","last_peered":"2024-09-11T14:43:49.504288+0000","last_clean":"2024-09-11T14:43:49.504288+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:43:49.504288+0000","last_undegraded":"2024-09-11T14:43:49.504288+0000","last_fullsized":"2024-09-11T14:43:49.504288+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":831,"log_dups_size":0,"ondisk_log_size":831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54460416,"num_objects":831,"num_object_clones":0,"num_object_copies":2493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":831,"num_write_kb":53184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"27'796","reported_seq":832,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:49.740917+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:43:49.740917+0000","last_peered":"2024-09-11T14:43:49.740917+0000","last_clean":"2024-09-11T14:43:49.740917+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:43:49.740917+0000","last_undegraded":"2024-09-11T14:43:49.740917+0000","last_fullsized":"2024-09-11T14:43:49.740917+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":796,"log_dups_size":0,"ondisk_log_size":796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52166656,"num_objects":796,"num_object_clones":0,"num_object_copies":2388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":796,"num_write_kb":50944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":121,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084543+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:42:13.084543+0000","last_peered":"2024-09-11T14:42:13.084543+0000","last_clean":"2024-09-11T14:42:13.084543+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:42:13.084543+0000","last_undegraded":"2024-09-11T14:42:13.084543+0000","last_fullsized":"2024-09-11T14:42:13.084543+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":50,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.184283+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:42:13.184283+0000","last_peered":"2024-09-11T14:42:13.184283+0000","last_clean":"2024-09-11T14:42:13.184283+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:42:13.184283+0000","last_undegraded":"2024-09-11T14:42:13.184283+0000","last_fullsized":"2024-09-11T14:42:13.184283+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"27'801","reported_seq":837,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:49.623831+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:43:49.623831+0000","last_peered":"2024-09-11T14:43:49.623831+0000","last_clean":"2024-09-11T14:43:49.623831+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:43:49.623831+0000","last_undegraded":"2024-09-11T14:43:49.623831+0000","last_fullsized":"2024-09-11T14:43:49.623831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":801,"log_dups_size":0,"ondisk_log_size":801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52494336,"num_objects":801,"num_object_clones":0,"num_object_copies":2403,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":801,"num_write_kb":51264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084333+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:42:13.084333+0000","last_peered":"2024-09-11T14:42:13.084333+0000","last_clean":"2024-09-11T14:42:13.084333+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:42:13.084333+0000","last_undegraded":"2024-09-11T14:42:13.084333+0000","last_fullsized":"2024-09-11T14:42:13.084333+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"27'809","reported_seq":845,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:49.322979+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:43:49.322979+0000","last_peered":"2024-09-11T14:43:49.322979+0000","last_clean":"2024-09-11T14:43:49.322979+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:43:49.322979+0000","last_undegraded":"2024-09-11T14:43:49.322979+0000","last_fullsized":"2024-09-11T14:43:49.322979+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":809,"log_dups_size":0,"ondisk_log_size":809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53018624,"num_objects":809,"num_object_clones":0,"num_object_copies":2427,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":809,"num_write_kb":51776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.179633+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:42:13.179633+0000","last_peered":"2024-09-11T14:42:13.179633+0000","last_clean":"2024-09-11T14:42:13.179633+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:42:13.179633+0000","last_undegraded":"2024-09-11T14:42:13.179633+0000","last_fullsized":"2024-09-11T14:42:13.179633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"27'810","reported_seq":846,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:49.787511+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:43:49.787511+0000","last_peered":"2024-09-11T14:43:49.787511+0000","last_clean":"2024-09-11T14:43:49.787511+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:43:49.787511+0000","last_undegraded":"2024-09-11T14:43:49.787511+0000","last_fullsized":"2024-09-11T14:43:49.787511+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":810,"log_dups_size":0,"ondisk_log_size":810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53084160,"num_objects":810,"num_object_clones":0,"num_object_copies":2430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":810,"num_write_kb":51840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":51,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.082423+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:42:13.082423+0000","last_peered":"2024-09-11T14:42:13.082423+0000","last_clean":"2024-09-11T14:42:13.082423+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:42:13.082423+0000","last_undegraded":"2024-09-11T14:42:13.082423+0000","last_fullsized":"2024-09-11T14:42:13.082423+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"27'799","reported_seq":835,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:50.645872+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:43:50.645872+0000","last_peered":"2024-09-11T14:43:50.645872+0000","last_clean":"2024-09-11T14:43:50.645872+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:43:50.645872+0000","last_undegraded":"2024-09-11T14:43:50.645872+0000","last_fullsized":"2024-09-11T14:43:50.645872+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":799,"log_dups_size":0,"ondisk_log_size":799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52363264,"num_objects":799,"num_object_clones":0,"num_object_copies":2397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":799,"num_write_kb":51136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"27'826","reported_seq":862,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:49.587867+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:43:49.587867+0000","last_peered":"2024-09-11T14:43:49.587867+0000","last_clean":"2024-09-11T14:43:49.587867+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:43:49.587867+0000","last_undegraded":"2024-09-11T14:43:49.587867+0000","last_fullsized":"2024-09-11T14:43:49.587867+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":826,"log_dups_size":0,"ondisk_log_size":826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54132736,"num_objects":826,"num_object_clones":0,"num_object_copies":2478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":826,"num_write_kb":52864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111507+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:42:13.111507+0000","last_peered":"2024-09-11T14:42:13.111507+0000","last_clean":"2024-09-11T14:42:13.111507+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:42:13.111507+0000","last_undegraded":"2024-09-11T14:42:13.111507+0000","last_fullsized":"2024-09-11T14:42:13.111507+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"27'752","reported_seq":788,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:50.385009+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:43:50.385009+0000","last_peered":"2024-09-11T14:43:50.385009+0000","last_clean":"2024-09-11T14:43:50.385009+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:43:50.385009+0000","last_undegraded":"2024-09-11T14:43:50.385009+0000","last_fullsized":"2024-09-11T14:43:50.385009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49283072,"num_objects":752,"num_object_clones":0,"num_object_copies":2256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":752,"num_write_kb":48128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.086961+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:42:13.086961+0000","last_peered":"2024-09-11T14:42:13.086961+0000","last_clean":"2024-09-11T14:42:13.086961+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:42:13.086961+0000","last_undegraded":"2024-09-11T14:42:13.086961+0000","last_fullsized":"2024-09-11T14:42:13.086961+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"27'819","reported_seq":855,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:49.485640+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:43:49.485640+0000","last_peered":"2024-09-11T14:43:49.485640+0000","last_clean":"2024-09-11T14:43:49.485640+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:43:49.485640+0000","last_undegraded":"2024-09-11T14:43:49.485640+0000","last_fullsized":"2024-09-11T14:43:49.485640+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":819,"log_dups_size":0,"ondisk_log_size":819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53673984,"num_objects":819,"num_object_clones":0,"num_object_copies":2457,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":819,"num_write_kb":52416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111171+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:42:13.111171+0000","last_peered":"2024-09-11T14:42:13.111171+0000","last_clean":"2024-09-11T14:42:13.111171+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:42:13.111171+0000","last_undegraded":"2024-09-11T14:42:13.111171+0000","last_fullsized":"2024-09-11T14:42:13.111171+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"27'814","reported_seq":850,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:50.722407+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:43:50.722407+0000","last_peered":"2024-09-11T14:43:50.722407+0000","last_clean":"2024-09-11T14:43:50.722407+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:43:50.722407+0000","last_undegraded":"2024-09-11T14:43:50.722407+0000","last_fullsized":"2024-09-11T14:43:50.722407+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":814,"log_dups_size":0,"ondisk_log_size":814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53346304,"num_objects":814,"num_object_clones":0,"num_object_copies":2442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":814,"num_write_kb":52096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.181385+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:42:13.181385+0000","last_peered":"2024-09-11T14:42:13.181385+0000","last_clean":"2024-09-11T14:42:13.181385+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:42:13.181385+0000","last_undegraded":"2024-09-11T14:42:13.181385+0000","last_fullsized":"2024-09-11T14:42:13.181385+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":854523904,"num_objects":13039,"num_object_clones":0,"num_object_copies":39117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13039,"num_write_kb":834496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":209033048064,"internally_reserved":0,"allocated":5178445824,"data_stored":5178445824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13039,"ondisk_log_size":13039,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":209033441280,"internally_reserved":0,"allocated":5178052608,"data_stored":5178052608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":52055977984,"internally_reserved":0,"allocated":1496895488,"data_stored":1496895488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673124,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1160972,"kb_used_data":1160972,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51136756,"statfs":{"total":53552873472,"available":52364038144,"internally_reserved":0,"allocated":1188835328,"data_stored":1188835328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705828,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1453504,"kb_used_data":1453504,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":50844224,"statfs":{"total":53552873472,"available":52064485376,"internally_reserved":0,"allocated":1488388096,"data_stored":1488388096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705828,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1272096,"kb_used_data":1272096,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51025632,"statfs":{"total":53552873472,"available":52250247168,"internally_reserved":0,"allocated":1302626304,"data_stored":1302626304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705828,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1145324,"kb_used_data":1145324,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51152404,"statfs":{"total":53552873472,"available":52380061696,"internally_reserved":0,"allocated":1172811776,"data_stored":1172811776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":52055977984,"internally_reserved":0,"allocated":1496895488,"data_stored":1496895488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":52374335488,"internally_reserved":0,"allocated":1178537984,"data_stored":1178537984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":52243386368,"internally_reserved":0,"allocated":1309487104,"data_stored":1309487104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":52359741440,"internally_reserved":0,"allocated":1193132032,"data_stored":1193132032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":52055977984,"internally_reserved":0,"allocated":1496895488,"data_stored":1496895488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":52374335488,"internally_reserved":0,"allocated":1178537984,"data_stored":1178537984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":52243386368,"internally_reserved":0,"allocated":1309487104,"data_stored":1309487104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":52359610368,"internally_reserved":0,"allocated":1193263104,"data_stored":1193263104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":52055715840,"internally_reserved":0,"allocated":1497157632,"data_stored":1497157632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:43:53.632 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:43:53.803 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:53.799+0000 7f459d151640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:53.809 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:53.808+0000 7f459d151640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:56.238 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:43:56.387 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:56.384+0000 7fd714f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:56.392 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:56.390+0000 7fd714f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:57.795 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:43:57.796 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:43:57.796 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:43:57.930 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:57.923+0000 7f0e7c991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:57.935 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:57.934+0000 7f0e7c991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:58.200 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:43:58.200 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:43:58.217 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2024-09-11T14:43:56.742607+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":838468171,"num_objects":12790,"num_object_clones":0,"num_object_copies":38370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":13503,"num_write_kb":839701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13477,"ondisk_log_size":13477,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":4997192,"kb_used_data":4997192,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":204193720,"statfs":{"total":214211493888,"available":209094369280,"internally_reserved":0,"allocated":5117124608,"data_stored":5117124608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31719448,"num_objects":485,"num_object_clones":0,"num_object_copies":1455,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1121,"num_write_kb":51329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002802"},"pg_stats":[{"pgid":"3.f","version":"27'816","reported_seq":852,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:53.780197+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:43:53.780197+0000","last_peered":"2024-09-11T14:43:53.780197+0000","last_clean":"2024-09-11T14:43:53.780197+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:43:53.780197+0000","last_undegraded":"2024-09-11T14:43:53.780197+0000","last_fullsized":"2024-09-11T14:43:53.780197+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":816,"log_dups_size":0,"ondisk_log_size":816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52559872,"num_objects":802,"num_object_clones":0,"num_object_copies":2406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":816,"num_write_kb":51776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"27'921","reported_seq":957,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:55.816561+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:43:55.816561+0000","last_peered":"2024-09-11T14:43:55.816561+0000","last_clean":"2024-09-11T14:43:55.816561+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:43:55.816561+0000","last_undegraded":"2024-09-11T14:43:55.816561+0000","last_fullsized":"2024-09-11T14:43:55.816561+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":921,"log_dups_size":0,"ondisk_log_size":921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56295424,"num_objects":859,"num_object_clones":0,"num_object_copies":2577,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":921,"num_write_kb":56960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"27'846","reported_seq":883,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:54.678713+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:43:54.678713+0000","last_peered":"2024-09-11T14:43:54.678713+0000","last_clean":"2024-09-11T14:43:54.678713+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:43:54.678713+0000","last_undegraded":"2024-09-11T14:43:54.678713+0000","last_fullsized":"2024-09-11T14:43:54.678713+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":846,"log_dups_size":0,"ondisk_log_size":846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53805080,"num_objects":822,"num_object_clones":0,"num_object_copies":2466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":846,"num_write_kb":53313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"27'815","reported_seq":851,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:54.450800+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:43:54.450800+0000","last_peered":"2024-09-11T14:43:54.450800+0000","last_clean":"2024-09-11T14:43:54.450800+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:43:54.450800+0000","last_undegraded":"2024-09-11T14:43:54.450800+0000","last_fullsized":"2024-09-11T14:43:54.450800+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":815,"log_dups_size":0,"ondisk_log_size":815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50135040,"num_objects":765,"num_object_clones":0,"num_object_copies":2295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":815,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"27'820","reported_seq":856,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:54.504172+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:43:54.504172+0000","last_peered":"2024-09-11T14:43:54.504172+0000","last_clean":"2024-09-11T14:43:54.504172+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:43:54.504172+0000","last_undegraded":"2024-09-11T14:43:54.504172+0000","last_fullsized":"2024-09-11T14:43:54.504172+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":820,"log_dups_size":0,"ondisk_log_size":820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51380224,"num_objects":784,"num_object_clones":0,"num_object_copies":2352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":820,"num_write_kb":51328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"27'906","reported_seq":942,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:54.404898+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:43:54.404898+0000","last_peered":"2024-09-11T14:43:54.404898+0000","last_clean":"2024-09-11T14:43:54.404898+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:43:54.404898+0000","last_undegraded":"2024-09-11T14:43:54.404898+0000","last_fullsized":"2024-09-11T14:43:54.404898+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":906,"log_dups_size":0,"ondisk_log_size":906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56492032,"num_objects":862,"num_object_clones":0,"num_object_copies":2586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":906,"num_write_kb":56576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"27'852","reported_seq":888,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:54.320577+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:43:54.320577+0000","last_peered":"2024-09-11T14:43:54.320577+0000","last_clean":"2024-09-11T14:43:54.320577+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:43:54.320577+0000","last_undegraded":"2024-09-11T14:43:54.320577+0000","last_fullsized":"2024-09-11T14:43:54.320577+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":852,"log_dups_size":0,"ondisk_log_size":852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53870592,"num_objects":822,"num_object_clones":0,"num_object_copies":2466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":852,"num_write_kb":53568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"27'811","reported_seq":847,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:54.692228+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:43:54.692228+0000","last_peered":"2024-09-11T14:43:54.692228+0000","last_clean":"2024-09-11T14:43:54.692228+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:43:54.692228+0000","last_undegraded":"2024-09-11T14:43:54.692228+0000","last_fullsized":"2024-09-11T14:43:54.692228+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":811,"log_dups_size":0,"ondisk_log_size":811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51314688,"num_objects":783,"num_object_clones":0,"num_object_copies":2349,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":811,"num_write_kb":51008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":121,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084543+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:42:13.084543+0000","last_peered":"2024-09-11T14:42:13.084543+0000","last_clean":"2024-09-11T14:42:13.084543+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:42:13.084543+0000","last_undegraded":"2024-09-11T14:42:13.084543+0000","last_fullsized":"2024-09-11T14:42:13.084543+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":50,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.184283+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:42:13.184283+0000","last_peered":"2024-09-11T14:42:13.184283+0000","last_clean":"2024-09-11T14:42:13.184283+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:42:13.184283+0000","last_undegraded":"2024-09-11T14:42:13.184283+0000","last_fullsized":"2024-09-11T14:42:13.184283+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"27'834","reported_seq":870,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:54.455255+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:43:54.455255+0000","last_peered":"2024-09-11T14:43:54.455255+0000","last_clean":"2024-09-11T14:43:54.455255+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:43:54.455255+0000","last_undegraded":"2024-09-11T14:43:54.455255+0000","last_fullsized":"2024-09-11T14:43:54.455255+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":834,"log_dups_size":0,"ondisk_log_size":834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51642368,"num_objects":788,"num_object_clones":0,"num_object_copies":2364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":834,"num_write_kb":51904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084333+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:42:13.084333+0000","last_peered":"2024-09-11T14:42:13.084333+0000","last_clean":"2024-09-11T14:42:13.084333+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:42:13.084333+0000","last_undegraded":"2024-09-11T14:42:13.084333+0000","last_fullsized":"2024-09-11T14:42:13.084333+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"27'831","reported_seq":867,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:54.337994+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:43:54.337994+0000","last_peered":"2024-09-11T14:43:54.337994+0000","last_clean":"2024-09-11T14:43:54.337994+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:43:54.337994+0000","last_undegraded":"2024-09-11T14:43:54.337994+0000","last_fullsized":"2024-09-11T14:43:54.337994+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":831,"log_dups_size":0,"ondisk_log_size":831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52363264,"num_objects":799,"num_object_clones":0,"num_object_copies":2397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":831,"num_write_kb":52160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.179633+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:42:13.179633+0000","last_peered":"2024-09-11T14:42:13.179633+0000","last_clean":"2024-09-11T14:42:13.179633+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:42:13.179633+0000","last_undegraded":"2024-09-11T14:42:13.179633+0000","last_fullsized":"2024-09-11T14:42:13.179633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"27'835","reported_seq":871,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:54.748107+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:43:54.748107+0000","last_peered":"2024-09-11T14:43:54.748107+0000","last_clean":"2024-09-11T14:43:54.748107+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:43:54.748107+0000","last_undegraded":"2024-09-11T14:43:54.748107+0000","last_fullsized":"2024-09-11T14:43:54.748107+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":835,"log_dups_size":0,"ondisk_log_size":835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51838976,"num_objects":791,"num_object_clones":0,"num_object_copies":2373,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":835,"num_write_kb":52032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":51,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.082423+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:42:13.082423+0000","last_peered":"2024-09-11T14:42:13.082423+0000","last_clean":"2024-09-11T14:42:13.082423+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:42:13.082423+0000","last_undegraded":"2024-09-11T14:42:13.082423+0000","last_fullsized":"2024-09-11T14:42:13.082423+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"27'824","reported_seq":860,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:55.661468+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:43:55.661468+0000","last_peered":"2024-09-11T14:43:55.661468+0000","last_clean":"2024-09-11T14:43:55.661468+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:43:55.661468+0000","last_undegraded":"2024-09-11T14:43:55.661468+0000","last_fullsized":"2024-09-11T14:43:55.661468+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":824,"log_dups_size":0,"ondisk_log_size":824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50724864,"num_objects":774,"num_object_clones":0,"num_object_copies":2322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":824,"num_write_kb":51136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"27'845","reported_seq":881,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:54.720485+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:43:54.720485+0000","last_peered":"2024-09-11T14:43:54.720485+0000","last_clean":"2024-09-11T14:43:54.720485+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:43:54.720485+0000","last_undegraded":"2024-09-11T14:43:54.720485+0000","last_fullsized":"2024-09-11T14:43:54.720485+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":845,"log_dups_size":0,"ondisk_log_size":845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53411840,"num_objects":815,"num_object_clones":0,"num_object_copies":2445,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":845,"num_write_kb":53120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111507+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:42:13.111507+0000","last_peered":"2024-09-11T14:42:13.111507+0000","last_clean":"2024-09-11T14:42:13.111507+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:42:13.111507+0000","last_undegraded":"2024-09-11T14:42:13.111507+0000","last_fullsized":"2024-09-11T14:42:13.111507+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"27'772","reported_seq":808,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:55.293839+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:43:55.293839+0000","last_peered":"2024-09-11T14:43:55.293839+0000","last_clean":"2024-09-11T14:43:55.293839+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:43:55.293839+0000","last_undegraded":"2024-09-11T14:43:55.293839+0000","last_fullsized":"2024-09-11T14:43:55.293839+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":772,"log_dups_size":0,"ondisk_log_size":772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47972352,"num_objects":732,"num_object_clones":0,"num_object_copies":2196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":772,"num_write_kb":48128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.086961+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:42:13.086961+0000","last_peered":"2024-09-11T14:42:13.086961+0000","last_clean":"2024-09-11T14:42:13.086961+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:42:13.086961+0000","last_undegraded":"2024-09-11T14:42:13.086961+0000","last_fullsized":"2024-09-11T14:42:13.086961+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"27'849","reported_seq":885,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:54.545869+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:43:54.545869+0000","last_peered":"2024-09-11T14:43:54.545869+0000","last_clean":"2024-09-11T14:43:54.545869+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:43:54.545869+0000","last_undegraded":"2024-09-11T14:43:54.545869+0000","last_fullsized":"2024-09-11T14:43:54.545869+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":849,"log_dups_size":0,"ondisk_log_size":849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52756480,"num_objects":805,"num_object_clones":0,"num_object_copies":2415,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":849,"num_write_kb":52928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111171+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:42:13.111171+0000","last_peered":"2024-09-11T14:42:13.111171+0000","last_clean":"2024-09-11T14:42:13.111171+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:42:13.111171+0000","last_undegraded":"2024-09-11T14:42:13.111171+0000","last_fullsized":"2024-09-11T14:42:13.111171+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"27'845","reported_seq":881,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:55.730971+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:43:55.730971+0000","last_peered":"2024-09-11T14:43:55.730971+0000","last_clean":"2024-09-11T14:43:55.730971+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:43:55.730971+0000","last_undegraded":"2024-09-11T14:43:55.730971+0000","last_fullsized":"2024-09-11T14:43:55.730971+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":845,"log_dups_size":0,"ondisk_log_size":845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51314688,"num_objects":783,"num_object_clones":0,"num_object_copies":2349,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":845,"num_write_kb":52096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.181385+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:42:13.181385+0000","last_peered":"2024-09-11T14:42:13.181385+0000","last_clean":"2024-09-11T14:42:13.181385+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:42:13.181385+0000","last_undegraded":"2024-09-11T14:42:13.181385+0000","last_fullsized":"2024-09-11T14:42:13.181385+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":837877784,"num_objects":12786,"num_object_clones":0,"num_object_copies":38358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13422,"num_write_kb":838593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":209124253696,"internally_reserved":0,"allocated":5087240192,"data_stored":5087240192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13422,"ondisk_log_size":13422,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":209123991552,"internally_reserved":0,"allocated":5087502336,"data_stored":5087502336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":52077015040,"internally_reserved":0,"allocated":1475858432,"data_stored":1475858432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673129,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1137256,"kb_used_data":1137256,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51160472,"statfs":{"total":53552873472,"available":52388323328,"internally_reserved":0,"allocated":1164550144,"data_stored":1164550144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705833,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1449076,"kb_used_data":1449076,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":50848652,"statfs":{"total":53552873472,"available":52069019648,"internally_reserved":0,"allocated":1483853824,"data_stored":1483853824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705833,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1269464,"kb_used_data":1269464,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51028264,"statfs":{"total":53552873472,"available":52252942336,"internally_reserved":0,"allocated":1299931136,"data_stored":1299931136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705833,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1141396,"kb_used_data":1141396,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51156332,"statfs":{"total":53552873472,"available":52384083968,"internally_reserved":0,"allocated":1168789504,"data_stored":1168789504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":52077015040,"internally_reserved":0,"allocated":1475858432,"data_stored":1475858432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":52391161856,"internally_reserved":0,"allocated":1161711616,"data_stored":1161711616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":52260806656,"internally_reserved":0,"allocated":1292066816,"data_stored":1292066816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":52394614784,"internally_reserved":0,"allocated":1158258688,"data_stored":1158258688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":52077408256,"internally_reserved":0,"allocated":1475465216,"data_stored":1475465216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":52391292928,"internally_reserved":0,"allocated":1161580544,"data_stored":1161580544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":52260806656,"internally_reserved":0,"allocated":1292066816,"data_stored":1292066816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":52394614784,"internally_reserved":0,"allocated":1158258688,"data_stored":1158258688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":52077539328,"internally_reserved":0,"allocated":1475334144,"data_stored":1475334144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:43:58.852 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:43:59.007 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:59.003+0000 7fb7a2093640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:59.012 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:43:59.011+0000 7fb7a2093640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:01.314 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:44:01.448 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:01.444+0000 7efe58ed1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:01.454 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:01.453+0000 7efe58ed1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:03.219 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:44:03.219 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:44:03.220 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:44:03.394 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:03.384+0000 7f6773791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:03.401 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:03.400+0000 7f6773791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:03.781 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:44:03.781 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:44:03.806 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":109,"stamp":"2024-09-11T14:44:02.744025+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":807993931,"num_objects":12325,"num_object_clones":0,"num_object_copies":36975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":13968,"num_write_kb":839701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13942,"ondisk_log_size":13942,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":4816808,"kb_used_data":4816808,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":204374104,"statfs":{"total":214211493888,"available":209279082496,"internally_reserved":0,"allocated":4932411392,"data_stored":4932411392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-41484264,"num_objects":-632,"num_object_clones":0,"num_object_copies":-1896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":934,"num_write_kb":9601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002785"},"pg_stats":[{"pgid":"3.f","version":"27'846","reported_seq":882,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:59.062334+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:43:59.062334+0000","last_peered":"2024-09-11T14:43:59.062334+0000","last_clean":"2024-09-11T14:43:59.062334+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:43:59.062334+0000","last_undegraded":"2024-09-11T14:43:59.062334+0000","last_fullsized":"2024-09-11T14:43:59.062334+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":846,"log_dups_size":0,"ondisk_log_size":846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50593792,"num_objects":772,"num_object_clones":0,"num_object_copies":2316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":846,"num_write_kb":51776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"27'960","reported_seq":996,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:00.454743+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:44:00.454743+0000","last_peered":"2024-09-11T14:44:00.454743+0000","last_clean":"2024-09-11T14:44:00.454743+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:44:00.454743+0000","last_undegraded":"2024-09-11T14:44:00.454743+0000","last_fullsized":"2024-09-11T14:44:00.454743+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":960,"log_dups_size":0,"ondisk_log_size":960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53739520,"num_objects":820,"num_object_clones":0,"num_object_copies":2460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":960,"num_write_kb":56960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"27'872","reported_seq":909,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:59.720022+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:43:59.720022+0000","last_peered":"2024-09-11T14:43:59.720022+0000","last_clean":"2024-09-11T14:43:59.720022+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:43:59.720022+0000","last_undegraded":"2024-09-11T14:43:59.720022+0000","last_fullsized":"2024-09-11T14:43:59.720022+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":872,"log_dups_size":0,"ondisk_log_size":872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52101144,"num_objects":796,"num_object_clones":0,"num_object_copies":2388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":872,"num_write_kb":53313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"27'844","reported_seq":880,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:59.232612+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:43:59.232612+0000","last_peered":"2024-09-11T14:43:59.232612+0000","last_clean":"2024-09-11T14:43:59.232612+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:43:59.232612+0000","last_undegraded":"2024-09-11T14:43:59.232612+0000","last_fullsized":"2024-09-11T14:43:59.232612+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":844,"log_dups_size":0,"ondisk_log_size":844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48234496,"num_objects":736,"num_object_clones":0,"num_object_copies":2208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":844,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"27'849","reported_seq":885,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:59.122971+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:43:59.122971+0000","last_peered":"2024-09-11T14:43:59.122971+0000","last_clean":"2024-09-11T14:43:59.122971+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:43:59.122971+0000","last_undegraded":"2024-09-11T14:43:59.122971+0000","last_fullsized":"2024-09-11T14:43:59.122971+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":849,"log_dups_size":0,"ondisk_log_size":849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49479680,"num_objects":755,"num_object_clones":0,"num_object_copies":2265,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":849,"num_write_kb":51328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"27'934","reported_seq":970,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:59.617293+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:43:59.617293+0000","last_peered":"2024-09-11T14:43:59.617293+0000","last_clean":"2024-09-11T14:43:59.617293+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:43:59.617293+0000","last_undegraded":"2024-09-11T14:43:59.617293+0000","last_fullsized":"2024-09-11T14:43:59.617293+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":934,"log_dups_size":0,"ondisk_log_size":934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54657024,"num_objects":834,"num_object_clones":0,"num_object_copies":2502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":934,"num_write_kb":56576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"27'876","reported_seq":912,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:59.775238+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:43:59.775238+0000","last_peered":"2024-09-11T14:43:59.775238+0000","last_clean":"2024-09-11T14:43:59.775238+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:43:59.775238+0000","last_undegraded":"2024-09-11T14:43:59.775238+0000","last_fullsized":"2024-09-11T14:43:59.775238+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":876,"log_dups_size":0,"ondisk_log_size":876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52297728,"num_objects":798,"num_object_clones":0,"num_object_copies":2394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":876,"num_write_kb":53568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"27'838","reported_seq":874,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:58.750407+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:43:58.750407+0000","last_peered":"2024-09-11T14:43:58.750407+0000","last_clean":"2024-09-11T14:43:58.750407+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:43:58.750407+0000","last_undegraded":"2024-09-11T14:43:58.750407+0000","last_fullsized":"2024-09-11T14:43:58.750407+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":838,"log_dups_size":0,"ondisk_log_size":838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49545216,"num_objects":756,"num_object_clones":0,"num_object_copies":2268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":838,"num_write_kb":51008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":121,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084543+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:42:13.084543+0000","last_peered":"2024-09-11T14:42:13.084543+0000","last_clean":"2024-09-11T14:42:13.084543+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:42:13.084543+0000","last_undegraded":"2024-09-11T14:42:13.084543+0000","last_fullsized":"2024-09-11T14:42:13.084543+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":50,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.184283+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:42:13.184283+0000","last_peered":"2024-09-11T14:42:13.184283+0000","last_clean":"2024-09-11T14:42:13.184283+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:42:13.184283+0000","last_undegraded":"2024-09-11T14:42:13.184283+0000","last_fullsized":"2024-09-11T14:42:13.184283+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"27'859","reported_seq":895,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:59.792666+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:43:59.792666+0000","last_peered":"2024-09-11T14:43:59.792666+0000","last_clean":"2024-09-11T14:43:59.792666+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:43:59.792666+0000","last_undegraded":"2024-09-11T14:43:59.792666+0000","last_fullsized":"2024-09-11T14:43:59.792666+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":859,"log_dups_size":0,"ondisk_log_size":859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50003968,"num_objects":763,"num_object_clones":0,"num_object_copies":2289,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":859,"num_write_kb":51904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084333+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:42:13.084333+0000","last_peered":"2024-09-11T14:42:13.084333+0000","last_clean":"2024-09-11T14:42:13.084333+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:42:13.084333+0000","last_undegraded":"2024-09-11T14:42:13.084333+0000","last_fullsized":"2024-09-11T14:42:13.084333+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"27'859","reported_seq":895,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:59.671631+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:43:59.671631+0000","last_peered":"2024-09-11T14:43:59.671631+0000","last_clean":"2024-09-11T14:43:59.671631+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:43:59.671631+0000","last_undegraded":"2024-09-11T14:43:59.671631+0000","last_fullsized":"2024-09-11T14:43:59.671631+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":859,"log_dups_size":0,"ondisk_log_size":859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50528256,"num_objects":771,"num_object_clones":0,"num_object_copies":2313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":859,"num_write_kb":52160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.179633+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:42:13.179633+0000","last_peered":"2024-09-11T14:42:13.179633+0000","last_clean":"2024-09-11T14:42:13.179633+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:42:13.179633+0000","last_undegraded":"2024-09-11T14:42:13.179633+0000","last_fullsized":"2024-09-11T14:42:13.179633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"27'871","reported_seq":907,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:59.477481+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:43:59.477481+0000","last_peered":"2024-09-11T14:43:59.477481+0000","last_clean":"2024-09-11T14:43:59.477481+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:43:59.477481+0000","last_undegraded":"2024-09-11T14:43:59.477481+0000","last_fullsized":"2024-09-11T14:43:59.477481+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":871,"log_dups_size":0,"ondisk_log_size":871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49479680,"num_objects":755,"num_object_clones":0,"num_object_copies":2265,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":871,"num_write_kb":52032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":51,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.082423+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:42:13.082423+0000","last_peered":"2024-09-11T14:42:13.082423+0000","last_clean":"2024-09-11T14:42:13.082423+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:42:13.082423+0000","last_undegraded":"2024-09-11T14:42:13.082423+0000","last_fullsized":"2024-09-11T14:42:13.082423+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"27'846","reported_seq":882,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:59.876354+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:43:59.876354+0000","last_peered":"2024-09-11T14:43:59.876354+0000","last_clean":"2024-09-11T14:43:59.876354+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:43:59.876354+0000","last_undegraded":"2024-09-11T14:43:59.876354+0000","last_fullsized":"2024-09-11T14:43:59.876354+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":846,"log_dups_size":0,"ondisk_log_size":846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49283072,"num_objects":752,"num_object_clones":0,"num_object_copies":2256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":846,"num_write_kb":51136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"27'879","reported_seq":915,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:59.775896+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:43:59.775896+0000","last_peered":"2024-09-11T14:43:59.775896+0000","last_clean":"2024-09-11T14:43:59.775896+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:43:59.775896+0000","last_undegraded":"2024-09-11T14:43:59.775896+0000","last_fullsized":"2024-09-11T14:43:59.775896+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":879,"log_dups_size":0,"ondisk_log_size":879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51183616,"num_objects":781,"num_object_clones":0,"num_object_copies":2343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":879,"num_write_kb":53120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111507+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:42:13.111507+0000","last_peered":"2024-09-11T14:42:13.111507+0000","last_clean":"2024-09-11T14:42:13.111507+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:42:13.111507+0000","last_undegraded":"2024-09-11T14:42:13.111507+0000","last_fullsized":"2024-09-11T14:42:13.111507+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"27'796","reported_seq":832,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:00.524075+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:44:00.524075+0000","last_peered":"2024-09-11T14:44:00.524075+0000","last_clean":"2024-09-11T14:44:00.524075+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:44:00.524075+0000","last_undegraded":"2024-09-11T14:44:00.524075+0000","last_fullsized":"2024-09-11T14:44:00.524075+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":796,"log_dups_size":0,"ondisk_log_size":796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46399488,"num_objects":708,"num_object_clones":0,"num_object_copies":2124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":796,"num_write_kb":48128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.086961+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:42:13.086961+0000","last_peered":"2024-09-11T14:42:13.086961+0000","last_clean":"2024-09-11T14:42:13.086961+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:42:13.086961+0000","last_undegraded":"2024-09-11T14:42:13.086961+0000","last_fullsized":"2024-09-11T14:42:13.086961+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"27'881","reported_seq":917,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:43:59.543877+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:43:59.543877+0000","last_peered":"2024-09-11T14:43:59.543877+0000","last_clean":"2024-09-11T14:43:59.543877+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:43:59.543877+0000","last_undegraded":"2024-09-11T14:43:59.543877+0000","last_fullsized":"2024-09-11T14:43:59.543877+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":881,"log_dups_size":0,"ondisk_log_size":881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50659328,"num_objects":773,"num_object_clones":0,"num_object_copies":2319,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":881,"num_write_kb":52928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111171+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:42:13.111171+0000","last_peered":"2024-09-11T14:42:13.111171+0000","last_clean":"2024-09-11T14:42:13.111171+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:42:13.111171+0000","last_undegraded":"2024-09-11T14:42:13.111171+0000","last_fullsized":"2024-09-11T14:42:13.111171+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"27'877","reported_seq":913,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:00.800124+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:44:00.800124+0000","last_peered":"2024-09-11T14:44:00.800124+0000","last_clean":"2024-09-11T14:44:00.800124+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:44:00.800124+0000","last_undegraded":"2024-09-11T14:44:00.800124+0000","last_fullsized":"2024-09-11T14:44:00.800124+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":877,"log_dups_size":0,"ondisk_log_size":877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49217536,"num_objects":751,"num_object_clones":0,"num_object_copies":2253,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":877,"num_write_kb":52096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.181385+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:42:13.181385+0000","last_peered":"2024-09-11T14:42:13.181385+0000","last_clean":"2024-09-11T14:42:13.181385+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:42:13.181385+0000","last_undegraded":"2024-09-11T14:42:13.181385+0000","last_fullsized":"2024-09-11T14:42:13.181385+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":807403544,"num_objects":12321,"num_object_clones":0,"num_object_copies":36963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13887,"num_write_kb":838593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":209308606464,"internally_reserved":0,"allocated":4902887424,"data_stored":4902887424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13887,"ondisk_log_size":13887,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":209308344320,"internally_reserved":0,"allocated":4903149568,"data_stored":4903149568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":52132569088,"internally_reserved":0,"allocated":1420304384,"data_stored":1420304384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673134,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1095072,"kb_used_data":1095072,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51202656,"statfs":{"total":53552873472,"available":52431519744,"internally_reserved":0,"allocated":1121353728,"data_stored":1121353728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705838,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1395540,"kb_used_data":1395540,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":50902188,"statfs":{"total":53552873472,"available":52123840512,"internally_reserved":0,"allocated":1429032960,"data_stored":1429032960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705838,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1223228,"kb_used_data":1223228,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51074500,"statfs":{"total":53552873472,"available":52300288000,"internally_reserved":0,"allocated":1252585472,"data_stored":1252585472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705838,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1102968,"kb_used_data":1102968,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51194760,"statfs":{"total":53552873472,"available":52423434240,"internally_reserved":0,"allocated":1129439232,"data_stored":1129439232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":52132569088,"internally_reserved":0,"allocated":1420304384,"data_stored":1420304384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":52430184448,"internally_reserved":0,"allocated":1122689024,"data_stored":1122689024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":52308094976,"internally_reserved":0,"allocated":1244778496,"data_stored":1244778496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":52437364736,"internally_reserved":0,"allocated":1115508736,"data_stored":1115508736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":52132700160,"internally_reserved":0,"allocated":1420173312,"data_stored":1420173312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":52430184448,"internally_reserved":0,"allocated":1122689024,"data_stored":1122689024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":52308357120,"internally_reserved":0,"allocated":1244516352,"data_stored":1244516352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":52437364736,"internally_reserved":0,"allocated":1115508736,"data_stored":1115508736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":52132700160,"internally_reserved":0,"allocated":1420173312,"data_stored":1420173312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:44:03.862 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:44:04.046 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:04.041+0000 7f81d1991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:04.051 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:04.049+0000 7f81d1991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:06.501 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:44:06.654 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:06.651+0000 7f744825c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:06.660 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:06.659+0000 7f744825c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:08.808 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:44:08.808 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:44:08.808 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:44:08.950 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:08.944+0000 7f8e42511640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:08.955 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:08.954+0000 7f8e42511640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:09.112 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:44:09.273 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:09.268+0000 7ff139cd1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:09.280 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:09.278+0000 7ff139cd1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:09.326 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:44:09.327 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:44:09.350 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2024-09-11T14:44:08.745501+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":776274507,"num_objects":11841,"num_object_clones":0,"num_object_copies":35523,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":14452,"num_write_kb":839701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14426,"ondisk_log_size":14426,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":4632632,"kb_used_data":4632632,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":204558280,"statfs":{"total":214211493888,"available":209467678720,"internally_reserved":0,"allocated":4743815168,"data_stored":4743815168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-62193664,"num_objects":-949,"num_object_clones":0,"num_object_copies":-2847,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":949,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002893"},"pg_stats":[{"pgid":"3.f","version":"27'877","reported_seq":913,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:04.388484+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:44:04.388484+0000","last_peered":"2024-09-11T14:44:04.388484+0000","last_clean":"2024-09-11T14:44:04.388484+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:44:04.388484+0000","last_undegraded":"2024-09-11T14:44:04.388484+0000","last_fullsized":"2024-09-11T14:44:04.388484+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":877,"log_dups_size":0,"ondisk_log_size":877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48562176,"num_objects":741,"num_object_clones":0,"num_object_copies":2223,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":877,"num_write_kb":51776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"27'986","reported_seq":1022,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:05.499212+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:44:05.499212+0000","last_peered":"2024-09-11T14:44:05.499212+0000","last_clean":"2024-09-11T14:44:05.499212+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:44:05.499212+0000","last_undegraded":"2024-09-11T14:44:05.499212+0000","last_fullsized":"2024-09-11T14:44:05.499212+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":986,"log_dups_size":0,"ondisk_log_size":986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52035584,"num_objects":794,"num_object_clones":0,"num_object_copies":2382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":986,"num_write_kb":56960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"27'914","reported_seq":951,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:04.740334+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:44:04.740334+0000","last_peered":"2024-09-11T14:44:04.740334+0000","last_clean":"2024-09-11T14:44:04.740334+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:44:04.740334+0000","last_undegraded":"2024-09-11T14:44:04.740334+0000","last_fullsized":"2024-09-11T14:44:04.740334+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":914,"log_dups_size":0,"ondisk_log_size":914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49348632,"num_objects":754,"num_object_clones":0,"num_object_copies":2262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":914,"num_write_kb":53313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"27'874","reported_seq":910,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:04.766431+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:44:04.766431+0000","last_peered":"2024-09-11T14:44:04.766431+0000","last_clean":"2024-09-11T14:44:04.766431+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:44:04.766431+0000","last_undegraded":"2024-09-11T14:44:04.766431+0000","last_fullsized":"2024-09-11T14:44:04.766431+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":874,"log_dups_size":0,"ondisk_log_size":874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46268416,"num_objects":706,"num_object_clones":0,"num_object_copies":2118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":874,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"27'876","reported_seq":912,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:04.573241+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:44:04.573241+0000","last_peered":"2024-09-11T14:44:04.573241+0000","last_clean":"2024-09-11T14:44:04.573241+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:44:04.573241+0000","last_undegraded":"2024-09-11T14:44:04.573241+0000","last_fullsized":"2024-09-11T14:44:04.573241+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":876,"log_dups_size":0,"ondisk_log_size":876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47710208,"num_objects":728,"num_object_clones":0,"num_object_copies":2184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":876,"num_write_kb":51328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"27'972","reported_seq":1008,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:04.662296+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:44:04.662296+0000","last_peered":"2024-09-11T14:44:04.662296+0000","last_clean":"2024-09-11T14:44:04.662296+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:44:04.662296+0000","last_undegraded":"2024-09-11T14:44:04.662296+0000","last_fullsized":"2024-09-11T14:44:04.662296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":972,"log_dups_size":0,"ondisk_log_size":972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52166656,"num_objects":796,"num_object_clones":0,"num_object_copies":2388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":972,"num_write_kb":56576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"27'908","reported_seq":944,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:04.278179+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:44:04.278179+0000","last_peered":"2024-09-11T14:44:04.278179+0000","last_clean":"2024-09-11T14:44:04.278179+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:44:04.278179+0000","last_undegraded":"2024-09-11T14:44:04.278179+0000","last_fullsized":"2024-09-11T14:44:04.278179+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":908,"log_dups_size":0,"ondisk_log_size":908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50200576,"num_objects":766,"num_object_clones":0,"num_object_copies":2298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":908,"num_write_kb":53568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"27'869","reported_seq":905,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:04.348305+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:44:04.348305+0000","last_peered":"2024-09-11T14:44:04.348305+0000","last_clean":"2024-09-11T14:44:04.348305+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:44:04.348305+0000","last_undegraded":"2024-09-11T14:44:04.348305+0000","last_fullsized":"2024-09-11T14:44:04.348305+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":869,"log_dups_size":0,"ondisk_log_size":869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47513600,"num_objects":725,"num_object_clones":0,"num_object_copies":2175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":51008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":121,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084543+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:42:13.084543+0000","last_peered":"2024-09-11T14:42:13.084543+0000","last_clean":"2024-09-11T14:42:13.084543+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:42:13.084543+0000","last_undegraded":"2024-09-11T14:42:13.084543+0000","last_fullsized":"2024-09-11T14:42:13.084543+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":50,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.184283+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:42:13.184283+0000","last_peered":"2024-09-11T14:42:13.184283+0000","last_clean":"2024-09-11T14:42:13.184283+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:42:13.184283+0000","last_undegraded":"2024-09-11T14:42:13.184283+0000","last_fullsized":"2024-09-11T14:42:13.184283+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"27'881","reported_seq":917,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:04.532170+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:44:04.532170+0000","last_peered":"2024-09-11T14:44:04.532170+0000","last_clean":"2024-09-11T14:44:04.532170+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:44:04.532170+0000","last_undegraded":"2024-09-11T14:44:04.532170+0000","last_fullsized":"2024-09-11T14:44:04.532170+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":881,"log_dups_size":0,"ondisk_log_size":881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48562176,"num_objects":741,"num_object_clones":0,"num_object_copies":2223,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":881,"num_write_kb":51904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084333+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:42:13.084333+0000","last_peered":"2024-09-11T14:42:13.084333+0000","last_clean":"2024-09-11T14:42:13.084333+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:42:13.084333+0000","last_undegraded":"2024-09-11T14:42:13.084333+0000","last_fullsized":"2024-09-11T14:42:13.084333+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"27'888","reported_seq":924,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:04.581930+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:44:04.581930+0000","last_peered":"2024-09-11T14:44:04.581930+0000","last_clean":"2024-09-11T14:44:04.581930+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:44:04.581930+0000","last_undegraded":"2024-09-11T14:44:04.581930+0000","last_fullsized":"2024-09-11T14:44:04.581930+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":888,"log_dups_size":0,"ondisk_log_size":888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48627712,"num_objects":742,"num_object_clones":0,"num_object_copies":2226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":888,"num_write_kb":52160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.179633+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:42:13.179633+0000","last_peered":"2024-09-11T14:42:13.179633+0000","last_clean":"2024-09-11T14:42:13.179633+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:42:13.179633+0000","last_undegraded":"2024-09-11T14:42:13.179633+0000","last_fullsized":"2024-09-11T14:42:13.179633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"27'910","reported_seq":946,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:04.417603+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:44:04.417603+0000","last_peered":"2024-09-11T14:44:04.417603+0000","last_clean":"2024-09-11T14:44:04.417603+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:44:04.417603+0000","last_undegraded":"2024-09-11T14:44:04.417603+0000","last_fullsized":"2024-09-11T14:44:04.417603+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":910,"log_dups_size":0,"ondisk_log_size":910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46923776,"num_objects":716,"num_object_clones":0,"num_object_copies":2148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":910,"num_write_kb":52032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":51,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.082423+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:42:13.082423+0000","last_peered":"2024-09-11T14:42:13.082423+0000","last_clean":"2024-09-11T14:42:13.082423+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:42:13.082423+0000","last_undegraded":"2024-09-11T14:42:13.082423+0000","last_fullsized":"2024-09-11T14:42:13.082423+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"27'870","reported_seq":906,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:05.552820+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:44:05.552820+0000","last_peered":"2024-09-11T14:44:05.552820+0000","last_clean":"2024-09-11T14:44:05.552820+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:44:05.552820+0000","last_undegraded":"2024-09-11T14:44:05.552820+0000","last_fullsized":"2024-09-11T14:44:05.552820+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":870,"log_dups_size":0,"ondisk_log_size":870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47710208,"num_objects":728,"num_object_clones":0,"num_object_copies":2184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":870,"num_write_kb":51136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"27'905","reported_seq":941,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:04.664779+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:44:04.664779+0000","last_peered":"2024-09-11T14:44:04.664779+0000","last_clean":"2024-09-11T14:44:04.664779+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:44:04.664779+0000","last_undegraded":"2024-09-11T14:44:04.664779+0000","last_fullsized":"2024-09-11T14:44:04.664779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":905,"log_dups_size":0,"ondisk_log_size":905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49479680,"num_objects":755,"num_object_clones":0,"num_object_copies":2265,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":905,"num_write_kb":53120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111507+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:42:13.111507+0000","last_peered":"2024-09-11T14:42:13.111507+0000","last_clean":"2024-09-11T14:42:13.111507+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:42:13.111507+0000","last_undegraded":"2024-09-11T14:42:13.111507+0000","last_fullsized":"2024-09-11T14:42:13.111507+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"27'823","reported_seq":859,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:05.693150+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:44:05.693150+0000","last_peered":"2024-09-11T14:44:05.693150+0000","last_clean":"2024-09-11T14:44:05.693150+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:44:05.693150+0000","last_undegraded":"2024-09-11T14:44:05.693150+0000","last_fullsized":"2024-09-11T14:44:05.693150+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":823,"log_dups_size":0,"ondisk_log_size":823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44630016,"num_objects":681,"num_object_clones":0,"num_object_copies":2043,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":823,"num_write_kb":48128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.086961+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:42:13.086961+0000","last_peered":"2024-09-11T14:42:13.086961+0000","last_clean":"2024-09-11T14:42:13.086961+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:42:13.086961+0000","last_undegraded":"2024-09-11T14:42:13.086961+0000","last_fullsized":"2024-09-11T14:42:13.086961+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"27'910","reported_seq":946,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:04.677014+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:44:04.677014+0000","last_peered":"2024-09-11T14:44:04.677014+0000","last_clean":"2024-09-11T14:44:04.677014+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:44:04.677014+0000","last_undegraded":"2024-09-11T14:44:04.677014+0000","last_fullsized":"2024-09-11T14:44:04.677014+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":910,"log_dups_size":0,"ondisk_log_size":910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48758784,"num_objects":744,"num_object_clones":0,"num_object_copies":2232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":910,"num_write_kb":52928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111171+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:42:13.111171+0000","last_peered":"2024-09-11T14:42:13.111171+0000","last_clean":"2024-09-11T14:42:13.111171+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:42:13.111171+0000","last_undegraded":"2024-09-11T14:42:13.111171+0000","last_fullsized":"2024-09-11T14:42:13.111171+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"27'908","reported_seq":944,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:05.447951+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:44:05.447951+0000","last_peered":"2024-09-11T14:44:05.447951+0000","last_clean":"2024-09-11T14:44:05.447951+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:44:05.447951+0000","last_undegraded":"2024-09-11T14:44:05.447951+0000","last_fullsized":"2024-09-11T14:44:05.447951+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":908,"log_dups_size":0,"ondisk_log_size":908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47185920,"num_objects":720,"num_object_clones":0,"num_object_copies":2160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":908,"num_write_kb":52096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.181385+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:42:13.181385+0000","last_peered":"2024-09-11T14:42:13.181385+0000","last_clean":"2024-09-11T14:42:13.181385+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:42:13.181385+0000","last_undegraded":"2024-09-11T14:42:13.181385+0000","last_fullsized":"2024-09-11T14:42:13.181385+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":775684120,"num_objects":11837,"num_object_clones":0,"num_object_copies":35511,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14371,"num_write_kb":838593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":209497124864,"internally_reserved":0,"allocated":4714369024,"data_stored":4714369024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14371,"ondisk_log_size":14371,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":209497124864,"internally_reserved":0,"allocated":4714369024,"data_stored":4714369024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":52185792512,"internally_reserved":0,"allocated":1367080960,"data_stored":1367080960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673139,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1053128,"kb_used_data":1053128,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51244600,"statfs":{"total":53552873472,"available":52474470400,"internally_reserved":0,"allocated":1078403072,"data_stored":1078403072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705843,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1343552,"kb_used_data":1343552,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":50954176,"statfs":{"total":53552873472,"available":52177076224,"internally_reserved":0,"allocated":1375797248,"data_stored":1375797248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705843,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1173336,"kb_used_data":1173336,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51124392,"statfs":{"total":53552873472,"available":52351377408,"internally_reserved":0,"allocated":1201496064,"data_stored":1201496064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705843,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1062616,"kb_used_data":1062616,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51235112,"statfs":{"total":53552873472,"available":52464754688,"internally_reserved":0,"allocated":1088118784,"data_stored":1088118784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":52185792512,"internally_reserved":0,"allocated":1367080960,"data_stored":1367080960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":52471504896,"internally_reserved":0,"allocated":1081368576,"data_stored":1081368576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":52358397952,"internally_reserved":0,"allocated":1194475520,"data_stored":1194475520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":52481429504,"internally_reserved":0,"allocated":1071443968,"data_stored":1071443968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":52185792512,"internally_reserved":0,"allocated":1367080960,"data_stored":1367080960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":52471504896,"internally_reserved":0,"allocated":1081368576,"data_stored":1081368576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":52358397952,"internally_reserved":0,"allocated":1194475520,"data_stored":1194475520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":52481429504,"internally_reserved":0,"allocated":1071443968,"data_stored":1071443968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":52185792512,"internally_reserved":0,"allocated":1367080960,"data_stored":1367080960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:44:11.817 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:44:11.968 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:11.965+0000 7f5660a7f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:11.973 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:11.971+0000 7f5660a7f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:14.294 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:44:14.353 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:44:14.353 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:44:14.353 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:44:14.429 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:14.425+0000 7fab72abd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:14.434 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:14.433+0000 7fab72abd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:14.519 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:14.513+0000 7ff2968d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:14.524 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:14.523+0000 7ff2968d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:14.902 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:44:14.902 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:44:14.919 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":115,"stamp":"2024-09-11T14:44:14.746765+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":745865803,"num_objects":11377,"num_object_clones":0,"num_object_copies":34131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":14916,"num_write_kb":839701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14890,"ondisk_log_size":14890,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":4457312,"kb_used_data":4457312,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":204733600,"statfs":{"total":214211493888,"available":209647206400,"internally_reserved":0,"allocated":4564287488,"data_stored":4564287488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-62128128,"num_objects":-948,"num_object_clones":0,"num_object_copies":-2844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":948,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002739"},"pg_stats":[{"pgid":"3.f","version":"27'902","reported_seq":938,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:09.709408+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:44:09.709408+0000","last_peered":"2024-09-11T14:44:09.709408+0000","last_clean":"2024-09-11T14:44:09.709408+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:44:09.709408+0000","last_undegraded":"2024-09-11T14:44:09.709408+0000","last_fullsized":"2024-09-11T14:44:09.709408+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":902,"log_dups_size":0,"ondisk_log_size":902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46923776,"num_objects":716,"num_object_clones":0,"num_object_copies":2148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":902,"num_write_kb":51776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"27'1012","reported_seq":1048,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:10.475034+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:44:10.475034+0000","last_peered":"2024-09-11T14:44:10.475034+0000","last_clean":"2024-09-11T14:44:10.475034+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:44:10.475034+0000","last_undegraded":"2024-09-11T14:44:10.475034+0000","last_fullsized":"2024-09-11T14:44:10.475034+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1012,"log_dups_size":0,"ondisk_log_size":1012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":768,"num_object_clones":0,"num_object_copies":2304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1012,"num_write_kb":56960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"27'942","reported_seq":979,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:09.673742+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:44:09.673742+0000","last_peered":"2024-09-11T14:44:09.673742+0000","last_clean":"2024-09-11T14:44:09.673742+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:44:09.673742+0000","last_undegraded":"2024-09-11T14:44:09.673742+0000","last_fullsized":"2024-09-11T14:44:09.673742+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":942,"log_dups_size":0,"ondisk_log_size":942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47513624,"num_objects":726,"num_object_clones":0,"num_object_copies":2178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":942,"num_write_kb":53313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"27'900","reported_seq":936,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:08.665641+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:44:08.665641+0000","last_peered":"2024-09-11T14:44:08.665641+0000","last_clean":"2024-09-11T14:44:08.665641+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:44:08.665641+0000","last_undegraded":"2024-09-11T14:44:08.665641+0000","last_fullsized":"2024-09-11T14:44:08.665641+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44564480,"num_objects":680,"num_object_clones":0,"num_object_copies":2040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":900,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"27'912","reported_seq":948,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:09.546163+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:44:09.546163+0000","last_peered":"2024-09-11T14:44:09.546163+0000","last_clean":"2024-09-11T14:44:09.546163+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:44:09.546163+0000","last_undegraded":"2024-09-11T14:44:09.546163+0000","last_fullsized":"2024-09-11T14:44:09.546163+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":912,"log_dups_size":0,"ondisk_log_size":912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45350912,"num_objects":692,"num_object_clones":0,"num_object_copies":2076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":912,"num_write_kb":51328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"27'1000","reported_seq":1036,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:09.576542+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:44:09.576542+0000","last_peered":"2024-09-11T14:44:09.576542+0000","last_clean":"2024-09-11T14:44:09.576542+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:44:09.576542+0000","last_undegraded":"2024-09-11T14:44:09.576542+0000","last_fullsized":"2024-09-11T14:44:09.576542+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1000,"log_dups_size":0,"ondisk_log_size":1000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":768,"num_object_clones":0,"num_object_copies":2304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1000,"num_write_kb":56576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"27'936","reported_seq":972,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:09.572755+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:44:09.572755+0000","last_peered":"2024-09-11T14:44:09.572755+0000","last_clean":"2024-09-11T14:44:09.572755+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:44:09.572755+0000","last_undegraded":"2024-09-11T14:44:09.572755+0000","last_fullsized":"2024-09-11T14:44:09.572755+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":936,"log_dups_size":0,"ondisk_log_size":936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48365568,"num_objects":738,"num_object_clones":0,"num_object_copies":2214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":936,"num_write_kb":53568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"27'890","reported_seq":926,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:09.368074+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:44:09.368074+0000","last_peered":"2024-09-11T14:44:09.368074+0000","last_clean":"2024-09-11T14:44:09.368074+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:44:09.368074+0000","last_undegraded":"2024-09-11T14:44:09.368074+0000","last_fullsized":"2024-09-11T14:44:09.368074+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":890,"log_dups_size":0,"ondisk_log_size":890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":704,"num_object_clones":0,"num_object_copies":2112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":890,"num_write_kb":51008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":121,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084543+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:42:13.084543+0000","last_peered":"2024-09-11T14:42:13.084543+0000","last_clean":"2024-09-11T14:42:13.084543+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:42:13.084543+0000","last_undegraded":"2024-09-11T14:42:13.084543+0000","last_fullsized":"2024-09-11T14:42:13.084543+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":50,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.184283+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:42:13.184283+0000","last_peered":"2024-09-11T14:42:13.184283+0000","last_clean":"2024-09-11T14:42:13.184283+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:42:13.184283+0000","last_undegraded":"2024-09-11T14:42:13.184283+0000","last_fullsized":"2024-09-11T14:42:13.184283+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"27'914","reported_seq":950,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:09.337828+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:44:09.337828+0000","last_peered":"2024-09-11T14:44:09.337828+0000","last_clean":"2024-09-11T14:44:09.337828+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:44:09.337828+0000","last_undegraded":"2024-09-11T14:44:09.337828+0000","last_fullsized":"2024-09-11T14:44:09.337828+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":914,"log_dups_size":0,"ondisk_log_size":914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46399488,"num_objects":708,"num_object_clones":0,"num_object_copies":2124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":914,"num_write_kb":51904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084333+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:42:13.084333+0000","last_peered":"2024-09-11T14:42:13.084333+0000","last_clean":"2024-09-11T14:42:13.084333+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:42:13.084333+0000","last_undegraded":"2024-09-11T14:42:13.084333+0000","last_fullsized":"2024-09-11T14:42:13.084333+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"27'915","reported_seq":951,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:09.416975+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:44:09.416975+0000","last_peered":"2024-09-11T14:44:09.416975+0000","last_clean":"2024-09-11T14:44:09.416975+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:44:09.416975+0000","last_undegraded":"2024-09-11T14:44:09.416975+0000","last_fullsized":"2024-09-11T14:44:09.416975+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":915,"log_dups_size":0,"ondisk_log_size":915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46858240,"num_objects":715,"num_object_clones":0,"num_object_copies":2145,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":915,"num_write_kb":52160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.179633+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:42:13.179633+0000","last_peered":"2024-09-11T14:42:13.179633+0000","last_clean":"2024-09-11T14:42:13.179633+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:42:13.179633+0000","last_undegraded":"2024-09-11T14:42:13.179633+0000","last_fullsized":"2024-09-11T14:42:13.179633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"27'945","reported_seq":981,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:09.624854+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:44:09.624854+0000","last_peered":"2024-09-11T14:44:09.624854+0000","last_clean":"2024-09-11T14:44:09.624854+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:44:09.624854+0000","last_undegraded":"2024-09-11T14:44:09.624854+0000","last_fullsized":"2024-09-11T14:44:09.624854+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":945,"log_dups_size":0,"ondisk_log_size":945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44630016,"num_objects":681,"num_object_clones":0,"num_object_copies":2043,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":945,"num_write_kb":52032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":51,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.082423+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:42:13.082423+0000","last_peered":"2024-09-11T14:42:13.082423+0000","last_clean":"2024-09-11T14:42:13.082423+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:42:13.082423+0000","last_undegraded":"2024-09-11T14:42:13.082423+0000","last_fullsized":"2024-09-11T14:42:13.082423+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"27'906","reported_seq":942,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:10.509698+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:44:10.509698+0000","last_peered":"2024-09-11T14:44:10.509698+0000","last_clean":"2024-09-11T14:44:10.509698+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:44:10.509698+0000","last_undegraded":"2024-09-11T14:44:10.509698+0000","last_fullsized":"2024-09-11T14:44:10.509698+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":906,"log_dups_size":0,"ondisk_log_size":906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45350912,"num_objects":692,"num_object_clones":0,"num_object_copies":2076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":906,"num_write_kb":51136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"27'934","reported_seq":970,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:09.558628+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:44:09.558628+0000","last_peered":"2024-09-11T14:44:09.558628+0000","last_clean":"2024-09-11T14:44:09.558628+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:44:09.558628+0000","last_undegraded":"2024-09-11T14:44:09.558628+0000","last_fullsized":"2024-09-11T14:44:09.558628+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":934,"log_dups_size":0,"ondisk_log_size":934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47579136,"num_objects":726,"num_object_clones":0,"num_object_copies":2178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":934,"num_write_kb":53120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111507+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:42:13.111507+0000","last_peered":"2024-09-11T14:42:13.111507+0000","last_clean":"2024-09-11T14:42:13.111507+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:42:13.111507+0000","last_undegraded":"2024-09-11T14:42:13.111507+0000","last_fullsized":"2024-09-11T14:42:13.111507+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"27'855","reported_seq":891,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:10.563287+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:44:10.563287+0000","last_peered":"2024-09-11T14:44:10.563287+0000","last_clean":"2024-09-11T14:44:10.563287+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:44:10.563287+0000","last_undegraded":"2024-09-11T14:44:10.563287+0000","last_fullsized":"2024-09-11T14:44:10.563287+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":855,"log_dups_size":0,"ondisk_log_size":855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42532864,"num_objects":649,"num_object_clones":0,"num_object_copies":1947,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":855,"num_write_kb":48128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.086961+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:42:13.086961+0000","last_peered":"2024-09-11T14:42:13.086961+0000","last_clean":"2024-09-11T14:42:13.086961+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:42:13.086961+0000","last_undegraded":"2024-09-11T14:42:13.086961+0000","last_fullsized":"2024-09-11T14:42:13.086961+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"27'942","reported_seq":978,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:09.694666+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:44:09.694666+0000","last_peered":"2024-09-11T14:44:09.694666+0000","last_clean":"2024-09-11T14:44:09.694666+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:44:09.694666+0000","last_undegraded":"2024-09-11T14:44:09.694666+0000","last_fullsized":"2024-09-11T14:44:09.694666+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":942,"log_dups_size":0,"ondisk_log_size":942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46661632,"num_objects":712,"num_object_clones":0,"num_object_copies":2136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":942,"num_write_kb":52928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111171+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:42:13.111171+0000","last_peered":"2024-09-11T14:42:13.111171+0000","last_clean":"2024-09-11T14:42:13.111171+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:42:13.111171+0000","last_undegraded":"2024-09-11T14:42:13.111171+0000","last_fullsized":"2024-09-11T14:42:13.111171+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"27'930","reported_seq":966,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:10.695311+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:44:10.695311+0000","last_peered":"2024-09-11T14:44:10.695311+0000","last_clean":"2024-09-11T14:44:10.695311+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:44:10.695311+0000","last_undegraded":"2024-09-11T14:44:10.695311+0000","last_fullsized":"2024-09-11T14:44:10.695311+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":930,"log_dups_size":0,"ondisk_log_size":930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45744128,"num_objects":698,"num_object_clones":0,"num_object_copies":2094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":930,"num_write_kb":52096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.181385+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:42:13.181385+0000","last_peered":"2024-09-11T14:42:13.181385+0000","last_clean":"2024-09-11T14:42:13.181385+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:42:13.181385+0000","last_undegraded":"2024-09-11T14:42:13.181385+0000","last_fullsized":"2024-09-11T14:42:13.181385+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":745275416,"num_objects":11373,"num_object_clones":0,"num_object_copies":34119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14835,"num_write_kb":838593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":209680334848,"internally_reserved":0,"allocated":4531159040,"data_stored":4531159040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14835,"ondisk_log_size":14835,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":209679941632,"internally_reserved":0,"allocated":4531552256,"data_stored":4531552256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":52239192064,"internally_reserved":0,"allocated":1313681408,"data_stored":1313681408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673144,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1012448,"kb_used_data":1012448,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51285280,"statfs":{"total":53552873472,"available":52516126720,"internally_reserved":0,"allocated":1036746752,"data_stored":1036746752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705848,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1292828,"kb_used_data":1292828,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51004900,"statfs":{"total":53552873472,"available":52229017600,"internally_reserved":0,"allocated":1323855872,"data_stored":1323855872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705848,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1132144,"kb_used_data":1132144,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51165584,"statfs":{"total":53552873472,"available":52393558016,"internally_reserved":0,"allocated":1159315456,"data_stored":1159315456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705848,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1019892,"kb_used_data":1019892,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51277836,"statfs":{"total":53552873472,"available":52508504064,"internally_reserved":0,"allocated":1044369408,"data_stored":1044369408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":52239192064,"internally_reserved":0,"allocated":1313681408,"data_stored":1313681408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":52515725312,"internally_reserved":0,"allocated":1037148160,"data_stored":1037148160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":52401754112,"internally_reserved":0,"allocated":1151119360,"data_stored":1151119360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":52523270144,"internally_reserved":0,"allocated":1029603328,"data_stored":1029603328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":52239192064,"internally_reserved":0,"allocated":1313681408,"data_stored":1313681408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":52515725312,"internally_reserved":0,"allocated":1037148160,"data_stored":1037148160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":52401885184,"internally_reserved":0,"allocated":1150988288,"data_stored":1150988288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":52523270144,"internally_reserved":0,"allocated":1029603328,"data_stored":1029603328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":52239454208,"internally_reserved":0,"allocated":1313419264,"data_stored":1313419264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:44:16.811 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:44:16.978 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:16.974+0000 7f5f21863640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:16.982 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:16.981+0000 7f5f21863640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:19.365 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:44:19.526 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:19.522+0000 7fb117591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:19.530 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:19.529+0000 7fb117591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:19.921 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:44:19.921 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:44:19.921 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:44:20.060 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:20.054+0000 7f32b6b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:20.066 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:20.064+0000 7f32b6b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:20.386 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:44:20.386 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:44:20.402 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":117,"stamp":"2024-09-11T14:44:18.747738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":714211915,"num_objects":10894,"num_object_clones":0,"num_object_copies":32682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":15399,"num_write_kb":839701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15373,"ondisk_log_size":15373,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":4266872,"kb_used_data":4266872,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":204924040,"statfs":{"total":214211493888,"available":209842216960,"internally_reserved":0,"allocated":4369276928,"data_stored":4369276928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-62062592,"num_objects":-947,"num_object_clones":0,"num_object_copies":-2841,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":947,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002800"},"pg_stats":[{"pgid":"3.f","version":"27'939","reported_seq":975,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:14.619859+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:44:14.619859+0000","last_peered":"2024-09-11T14:44:14.619859+0000","last_clean":"2024-09-11T14:44:14.619859+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:44:14.619859+0000","last_undegraded":"2024-09-11T14:44:14.619859+0000","last_fullsized":"2024-09-11T14:44:14.619859+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":939,"log_dups_size":0,"ondisk_log_size":939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44498944,"num_objects":679,"num_object_clones":0,"num_object_copies":2037,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":939,"num_write_kb":51776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"27'1053","reported_seq":1089,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:15.611241+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:44:15.611241+0000","last_peered":"2024-09-11T14:44:15.611241+0000","last_clean":"2024-09-11T14:44:15.611241+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:44:15.611241+0000","last_undegraded":"2024-09-11T14:44:15.611241+0000","last_fullsized":"2024-09-11T14:44:15.611241+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1053,"log_dups_size":0,"ondisk_log_size":1053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47644672,"num_objects":727,"num_object_clones":0,"num_object_copies":2181,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1053,"num_write_kb":56960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"27'974","reported_seq":1011,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:14.795806+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:44:14.795806+0000","last_peered":"2024-09-11T14:44:14.795806+0000","last_clean":"2024-09-11T14:44:14.795806+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:44:14.795806+0000","last_undegraded":"2024-09-11T14:44:14.795806+0000","last_fullsized":"2024-09-11T14:44:14.795806+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":974,"log_dups_size":0,"ondisk_log_size":974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45416472,"num_objects":694,"num_object_clones":0,"num_object_copies":2082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":974,"num_write_kb":53313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"27'928","reported_seq":964,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:14.167470+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:44:14.167470+0000","last_peered":"2024-09-11T14:44:14.167470+0000","last_clean":"2024-09-11T14:44:14.167470+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:44:14.167470+0000","last_undegraded":"2024-09-11T14:44:14.167470+0000","last_fullsized":"2024-09-11T14:44:14.167470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":928,"log_dups_size":0,"ondisk_log_size":928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42729472,"num_objects":652,"num_object_clones":0,"num_object_copies":1956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":928,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"27'941","reported_seq":977,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:14.698958+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:44:14.698958+0000","last_peered":"2024-09-11T14:44:14.698958+0000","last_clean":"2024-09-11T14:44:14.698958+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:44:14.698958+0000","last_undegraded":"2024-09-11T14:44:14.698958+0000","last_fullsized":"2024-09-11T14:44:14.698958+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":941,"log_dups_size":0,"ondisk_log_size":941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43450368,"num_objects":663,"num_object_clones":0,"num_object_copies":1989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":941,"num_write_kb":51328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"27'1033","reported_seq":1069,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:14.454422+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:44:14.454422+0000","last_peered":"2024-09-11T14:44:14.454422+0000","last_clean":"2024-09-11T14:44:14.454422+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:44:14.454422+0000","last_undegraded":"2024-09-11T14:44:14.454422+0000","last_fullsized":"2024-09-11T14:44:14.454422+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1033,"log_dups_size":0,"ondisk_log_size":1033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48168960,"num_objects":735,"num_object_clones":0,"num_object_copies":2205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1033,"num_write_kb":56576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"27'971","reported_seq":1007,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:14.811241+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:44:14.811241+0000","last_peered":"2024-09-11T14:44:14.811241+0000","last_clean":"2024-09-11T14:44:14.811241+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:44:14.811241+0000","last_undegraded":"2024-09-11T14:44:14.811241+0000","last_fullsized":"2024-09-11T14:44:14.811241+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":971,"log_dups_size":0,"ondisk_log_size":971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46071808,"num_objects":703,"num_object_clones":0,"num_object_copies":2109,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":971,"num_write_kb":53568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"27'931","reported_seq":967,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:14.691391+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:44:14.691391+0000","last_peered":"2024-09-11T14:44:14.691391+0000","last_clean":"2024-09-11T14:44:14.691391+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:44:14.691391+0000","last_undegraded":"2024-09-11T14:44:14.691391+0000","last_fullsized":"2024-09-11T14:44:14.691391+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":931,"log_dups_size":0,"ondisk_log_size":931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43450368,"num_objects":663,"num_object_clones":0,"num_object_copies":1989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":931,"num_write_kb":51008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":121,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084543+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:42:13.084543+0000","last_peered":"2024-09-11T14:42:13.084543+0000","last_clean":"2024-09-11T14:42:13.084543+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:42:13.084543+0000","last_undegraded":"2024-09-11T14:42:13.084543+0000","last_fullsized":"2024-09-11T14:42:13.084543+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":50,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.184283+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:42:13.184283+0000","last_peered":"2024-09-11T14:42:13.184283+0000","last_clean":"2024-09-11T14:42:13.184283+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:42:13.184283+0000","last_undegraded":"2024-09-11T14:42:13.184283+0000","last_fullsized":"2024-09-11T14:42:13.184283+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"27'939","reported_seq":975,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:14.226305+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:44:14.226305+0000","last_peered":"2024-09-11T14:44:14.226305+0000","last_clean":"2024-09-11T14:44:14.226305+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:44:14.226305+0000","last_undegraded":"2024-09-11T14:44:14.226305+0000","last_fullsized":"2024-09-11T14:44:14.226305+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":939,"log_dups_size":0,"ondisk_log_size":939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44761088,"num_objects":683,"num_object_clones":0,"num_object_copies":2049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":939,"num_write_kb":51904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084333+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:42:13.084333+0000","last_peered":"2024-09-11T14:42:13.084333+0000","last_clean":"2024-09-11T14:42:13.084333+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:42:13.084333+0000","last_undegraded":"2024-09-11T14:42:13.084333+0000","last_fullsized":"2024-09-11T14:42:13.084333+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"27'936","reported_seq":972,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:14.568233+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:44:14.568233+0000","last_peered":"2024-09-11T14:44:14.568233+0000","last_clean":"2024-09-11T14:44:14.568233+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:44:14.568233+0000","last_undegraded":"2024-09-11T14:44:14.568233+0000","last_fullsized":"2024-09-11T14:44:14.568233+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":936,"log_dups_size":0,"ondisk_log_size":936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45481984,"num_objects":694,"num_object_clones":0,"num_object_copies":2082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":936,"num_write_kb":52160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.179633+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:42:13.179633+0000","last_peered":"2024-09-11T14:42:13.179633+0000","last_clean":"2024-09-11T14:42:13.179633+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:42:13.179633+0000","last_undegraded":"2024-09-11T14:42:13.179633+0000","last_fullsized":"2024-09-11T14:42:13.179633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"27'973","reported_seq":1009,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:13.260060+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:44:13.260060+0000","last_peered":"2024-09-11T14:44:13.260060+0000","last_clean":"2024-09-11T14:44:13.260060+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:44:13.260060+0000","last_undegraded":"2024-09-11T14:44:13.260060+0000","last_fullsized":"2024-09-11T14:44:13.260060+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":973,"log_dups_size":0,"ondisk_log_size":973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42795008,"num_objects":653,"num_object_clones":0,"num_object_copies":1959,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":973,"num_write_kb":52032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":51,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.082423+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:42:13.082423+0000","last_peered":"2024-09-11T14:42:13.082423+0000","last_clean":"2024-09-11T14:42:13.082423+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:42:13.082423+0000","last_undegraded":"2024-09-11T14:42:13.082423+0000","last_fullsized":"2024-09-11T14:42:13.082423+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"27'935","reported_seq":971,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:15.597385+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:44:15.597385+0000","last_peered":"2024-09-11T14:44:15.597385+0000","last_clean":"2024-09-11T14:44:15.597385+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:44:15.597385+0000","last_undegraded":"2024-09-11T14:44:15.597385+0000","last_fullsized":"2024-09-11T14:44:15.597385+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":935,"log_dups_size":0,"ondisk_log_size":935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43450368,"num_objects":663,"num_object_clones":0,"num_object_copies":1989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":935,"num_write_kb":51136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"27'962","reported_seq":998,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:14.413642+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:44:14.413642+0000","last_peered":"2024-09-11T14:44:14.413642+0000","last_clean":"2024-09-11T14:44:14.413642+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:44:14.413642+0000","last_undegraded":"2024-09-11T14:44:14.413642+0000","last_fullsized":"2024-09-11T14:44:14.413642+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":962,"log_dups_size":0,"ondisk_log_size":962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45744128,"num_objects":698,"num_object_clones":0,"num_object_copies":2094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":962,"num_write_kb":53120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111507+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:42:13.111507+0000","last_peered":"2024-09-11T14:42:13.111507+0000","last_clean":"2024-09-11T14:42:13.111507+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:42:13.111507+0000","last_undegraded":"2024-09-11T14:42:13.111507+0000","last_fullsized":"2024-09-11T14:42:13.111507+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"27'876","reported_seq":912,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:15.355580+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:44:15.355580+0000","last_peered":"2024-09-11T14:44:15.355580+0000","last_clean":"2024-09-11T14:44:15.355580+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:44:15.355580+0000","last_undegraded":"2024-09-11T14:44:15.355580+0000","last_fullsized":"2024-09-11T14:44:15.355580+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":876,"log_dups_size":0,"ondisk_log_size":876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41156608,"num_objects":628,"num_object_clones":0,"num_object_copies":1884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":876,"num_write_kb":48128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.086961+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:42:13.086961+0000","last_peered":"2024-09-11T14:42:13.086961+0000","last_clean":"2024-09-11T14:42:13.086961+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:42:13.086961+0000","last_undegraded":"2024-09-11T14:42:13.086961+0000","last_fullsized":"2024-09-11T14:42:13.086961+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"27'970","reported_seq":1006,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:14.472442+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:44:14.472442+0000","last_peered":"2024-09-11T14:44:14.472442+0000","last_clean":"2024-09-11T14:44:14.472442+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:44:14.472442+0000","last_undegraded":"2024-09-11T14:44:14.472442+0000","last_fullsized":"2024-09-11T14:44:14.472442+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":970,"log_dups_size":0,"ondisk_log_size":970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44826624,"num_objects":684,"num_object_clones":0,"num_object_copies":2052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":970,"num_write_kb":52928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111171+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:42:13.111171+0000","last_peered":"2024-09-11T14:42:13.111171+0000","last_clean":"2024-09-11T14:42:13.111171+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:42:13.111171+0000","last_undegraded":"2024-09-11T14:42:13.111171+0000","last_fullsized":"2024-09-11T14:42:13.111171+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"27'957","reported_seq":993,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:15.676005+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:44:15.676005+0000","last_peered":"2024-09-11T14:44:15.676005+0000","last_clean":"2024-09-11T14:44:15.676005+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:44:15.676005+0000","last_undegraded":"2024-09-11T14:44:15.676005+0000","last_fullsized":"2024-09-11T14:44:15.676005+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":957,"log_dups_size":0,"ondisk_log_size":957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43974656,"num_objects":671,"num_object_clones":0,"num_object_copies":2013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":957,"num_write_kb":52096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.181385+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:42:13.181385+0000","last_peered":"2024-09-11T14:42:13.181385+0000","last_clean":"2024-09-11T14:42:13.181385+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:42:13.181385+0000","last_undegraded":"2024-09-11T14:42:13.181385+0000","last_fullsized":"2024-09-11T14:42:13.181385+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":713621528,"num_objects":10890,"num_object_clones":0,"num_object_copies":32670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15318,"num_write_kb":838593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":209868996608,"internally_reserved":0,"allocated":4342497280,"data_stored":4342497280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15318,"ondisk_log_size":15318,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":209868865536,"internally_reserved":0,"allocated":4342628352,"data_stored":4342628352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":52294541312,"internally_reserved":0,"allocated":1258332160,"data_stored":1258332160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":968964,"kb_used_data":968964,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51328764,"statfs":{"total":53552873472,"available":52560654336,"internally_reserved":0,"allocated":992219136,"data_stored":992219136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1236776,"kb_used_data":1236776,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51060952,"statfs":{"total":53552873472,"available":52286414848,"internally_reserved":0,"allocated":1266458624,"data_stored":1266458624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1083856,"kb_used_data":1083856,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51213872,"statfs":{"total":53552873472,"available":52443004928,"internally_reserved":0,"allocated":1109868544,"data_stored":1109868544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":977276,"kb_used_data":977276,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51320452,"statfs":{"total":53552873472,"available":52552142848,"internally_reserved":0,"allocated":1000730624,"data_stored":1000730624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":52294541312,"internally_reserved":0,"allocated":1258332160,"data_stored":1258332160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":52557803520,"internally_reserved":0,"allocated":995069952,"data_stored":995069952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":52449828864,"internally_reserved":0,"allocated":1103044608,"data_stored":1103044608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":52566560768,"internally_reserved":0,"allocated":986312704,"data_stored":986312704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":52294672384,"internally_reserved":0,"allocated":1258201088,"data_stored":1258201088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":52557803520,"internally_reserved":0,"allocated":995069952,"data_stored":995069952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":52449959936,"internally_reserved":0,"allocated":1102913536,"data_stored":1102913536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":52566560768,"internally_reserved":0,"allocated":986312704,"data_stored":986312704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":52294672384,"internally_reserved":0,"allocated":1258201088,"data_stored":1258201088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:44:21.946 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:44:22.106 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:22.102+0000 7f6dcaee3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:22.113 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:22.112+0000 7f6dcaee3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:24.517 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:44:24.662 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:24.658+0000 7f890a391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:24.666 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:24.665+0000 7f890a391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:25.405 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:44:25.405 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:44:25.405 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:44:25.575 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:25.567+0000 7f46780d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:25.581 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:25.580+0000 7f46780d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:25.847 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:44:25.848 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:44:25.864 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2024-09-11T14:44:24.749225+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":683344459,"num_objects":10423,"num_object_clones":0,"num_object_copies":31269,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":15870,"num_write_kb":839701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15844,"ondisk_log_size":15844,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":4088696,"kb_used_data":4088696,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":205102216,"statfs":{"total":214211493888,"available":210024669184,"internally_reserved":0,"allocated":4186824704,"data_stored":4186824704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-62521344,"num_objects":-954,"num_object_clones":0,"num_object_copies":-2862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":954,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002873"},"pg_stats":[{"pgid":"3.f","version":"27'957","reported_seq":993,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:18.842330+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:44:18.842330+0000","last_peered":"2024-09-11T14:44:18.842330+0000","last_clean":"2024-09-11T14:44:18.842330+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:44:18.842330+0000","last_undegraded":"2024-09-11T14:44:18.842330+0000","last_fullsized":"2024-09-11T14:44:18.842330+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":957,"log_dups_size":0,"ondisk_log_size":957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43319296,"num_objects":661,"num_object_clones":0,"num_object_copies":1983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":957,"num_write_kb":51776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"27'1086","reported_seq":1122,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:20.739993+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:44:20.739993+0000","last_peered":"2024-09-11T14:44:20.739993+0000","last_clean":"2024-09-11T14:44:20.739993+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:44:20.739993+0000","last_undegraded":"2024-09-11T14:44:20.739993+0000","last_fullsized":"2024-09-11T14:44:20.739993+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1086,"log_dups_size":0,"ondisk_log_size":1086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45481984,"num_objects":694,"num_object_clones":0,"num_object_copies":2082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1086,"num_write_kb":56960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"27'997","reported_seq":1034,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:19.404534+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:44:19.404534+0000","last_peered":"2024-09-11T14:44:19.404534+0000","last_clean":"2024-09-11T14:44:19.404534+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:44:19.404534+0000","last_undegraded":"2024-09-11T14:44:19.404534+0000","last_fullsized":"2024-09-11T14:44:19.404534+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":997,"log_dups_size":0,"ondisk_log_size":997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43909144,"num_objects":671,"num_object_clones":0,"num_object_copies":2013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":997,"num_write_kb":53313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"27'966","reported_seq":1002,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:19.367166+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:44:19.367166+0000","last_peered":"2024-09-11T14:44:19.367166+0000","last_clean":"2024-09-11T14:44:19.367166+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:44:19.367166+0000","last_undegraded":"2024-09-11T14:44:19.367166+0000","last_fullsized":"2024-09-11T14:44:19.367166+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":966,"log_dups_size":0,"ondisk_log_size":966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40239104,"num_objects":614,"num_object_clones":0,"num_object_copies":1842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":966,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"27'974","reported_seq":1010,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:19.547737+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:44:19.547737+0000","last_peered":"2024-09-11T14:44:19.547737+0000","last_clean":"2024-09-11T14:44:19.547737+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:44:19.547737+0000","last_undegraded":"2024-09-11T14:44:19.547737+0000","last_fullsized":"2024-09-11T14:44:19.547737+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":974,"log_dups_size":0,"ondisk_log_size":974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41287680,"num_objects":630,"num_object_clones":0,"num_object_copies":1890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":974,"num_write_kb":51328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"27'1056","reported_seq":1092,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:19.486591+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:44:19.486591+0000","last_peered":"2024-09-11T14:44:19.486591+0000","last_clean":"2024-09-11T14:44:19.486591+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:44:19.486591+0000","last_undegraded":"2024-09-11T14:44:19.486591+0000","last_fullsized":"2024-09-11T14:44:19.486591+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1056,"log_dups_size":0,"ondisk_log_size":1056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46661632,"num_objects":712,"num_object_clones":0,"num_object_copies":2136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1056,"num_write_kb":56576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"27'1002","reported_seq":1038,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:19.405774+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:44:19.405774+0000","last_peered":"2024-09-11T14:44:19.405774+0000","last_clean":"2024-09-11T14:44:19.405774+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:44:19.405774+0000","last_undegraded":"2024-09-11T14:44:19.405774+0000","last_fullsized":"2024-09-11T14:44:19.405774+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1002,"log_dups_size":0,"ondisk_log_size":1002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44040192,"num_objects":672,"num_object_clones":0,"num_object_copies":2016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1002,"num_write_kb":53568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"27'962","reported_seq":998,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:19.535857+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:44:19.535857+0000","last_peered":"2024-09-11T14:44:19.535857+0000","last_clean":"2024-09-11T14:44:19.535857+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:44:19.535857+0000","last_undegraded":"2024-09-11T14:44:19.535857+0000","last_fullsized":"2024-09-11T14:44:19.535857+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":962,"log_dups_size":0,"ondisk_log_size":962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41418752,"num_objects":632,"num_object_clones":0,"num_object_copies":1896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":962,"num_write_kb":51008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":121,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084543+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:42:13.084543+0000","last_peered":"2024-09-11T14:42:13.084543+0000","last_clean":"2024-09-11T14:42:13.084543+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:42:13.084543+0000","last_undegraded":"2024-09-11T14:42:13.084543+0000","last_fullsized":"2024-09-11T14:42:13.084543+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":50,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.184283+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:42:13.184283+0000","last_peered":"2024-09-11T14:42:13.184283+0000","last_clean":"2024-09-11T14:42:13.184283+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:42:13.184283+0000","last_undegraded":"2024-09-11T14:42:13.184283+0000","last_fullsized":"2024-09-11T14:42:13.184283+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"27'974","reported_seq":1010,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:19.663591+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:44:19.663591+0000","last_peered":"2024-09-11T14:44:19.663591+0000","last_clean":"2024-09-11T14:44:19.663591+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:44:19.663591+0000","last_undegraded":"2024-09-11T14:44:19.663591+0000","last_fullsized":"2024-09-11T14:44:19.663591+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":974,"log_dups_size":0,"ondisk_log_size":974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42467328,"num_objects":648,"num_object_clones":0,"num_object_copies":1944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":974,"num_write_kb":51904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084333+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:42:13.084333+0000","last_peered":"2024-09-11T14:42:13.084333+0000","last_clean":"2024-09-11T14:42:13.084333+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:42:13.084333+0000","last_undegraded":"2024-09-11T14:42:13.084333+0000","last_fullsized":"2024-09-11T14:42:13.084333+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"27'960","reported_seq":996,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:19.536467+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:44:19.536467+0000","last_peered":"2024-09-11T14:44:19.536467+0000","last_clean":"2024-09-11T14:44:19.536467+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:44:19.536467+0000","last_undegraded":"2024-09-11T14:44:19.536467+0000","last_fullsized":"2024-09-11T14:44:19.536467+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":960,"log_dups_size":0,"ondisk_log_size":960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43909120,"num_objects":670,"num_object_clones":0,"num_object_copies":2010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":960,"num_write_kb":52160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.179633+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:42:13.179633+0000","last_peered":"2024-09-11T14:42:13.179633+0000","last_clean":"2024-09-11T14:42:13.179633+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:42:13.179633+0000","last_undegraded":"2024-09-11T14:42:13.179633+0000","last_fullsized":"2024-09-11T14:42:13.179633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"27'999","reported_seq":1035,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:19.616955+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:44:19.616955+0000","last_peered":"2024-09-11T14:44:19.616955+0000","last_clean":"2024-09-11T14:44:19.616955+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:44:19.616955+0000","last_undegraded":"2024-09-11T14:44:19.616955+0000","last_fullsized":"2024-09-11T14:44:19.616955+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":999,"log_dups_size":0,"ondisk_log_size":999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41091072,"num_objects":627,"num_object_clones":0,"num_object_copies":1881,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":999,"num_write_kb":52032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":51,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.082423+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:42:13.082423+0000","last_peered":"2024-09-11T14:42:13.082423+0000","last_clean":"2024-09-11T14:42:13.082423+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:42:13.082423+0000","last_undegraded":"2024-09-11T14:42:13.082423+0000","last_fullsized":"2024-09-11T14:42:13.082423+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"27'963","reported_seq":999,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:20.573525+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:44:20.573525+0000","last_peered":"2024-09-11T14:44:20.573525+0000","last_clean":"2024-09-11T14:44:20.573525+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:44:20.573525+0000","last_undegraded":"2024-09-11T14:44:20.573525+0000","last_fullsized":"2024-09-11T14:44:20.573525+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":963,"log_dups_size":0,"ondisk_log_size":963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41615360,"num_objects":635,"num_object_clones":0,"num_object_copies":1905,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":963,"num_write_kb":51136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"27'998","reported_seq":1034,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:19.578129+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:44:19.578129+0000","last_peered":"2024-09-11T14:44:19.578129+0000","last_clean":"2024-09-11T14:44:19.578129+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:44:19.578129+0000","last_undegraded":"2024-09-11T14:44:19.578129+0000","last_fullsized":"2024-09-11T14:44:19.578129+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":998,"log_dups_size":0,"ondisk_log_size":998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43384832,"num_objects":662,"num_object_clones":0,"num_object_copies":1986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":998,"num_write_kb":53120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111507+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:42:13.111507+0000","last_peered":"2024-09-11T14:42:13.111507+0000","last_clean":"2024-09-11T14:42:13.111507+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:42:13.111507+0000","last_undegraded":"2024-09-11T14:42:13.111507+0000","last_fullsized":"2024-09-11T14:42:13.111507+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"27'908","reported_seq":944,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:20.506789+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:44:20.506789+0000","last_peered":"2024-09-11T14:44:20.506789+0000","last_clean":"2024-09-11T14:44:20.506789+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:44:20.506789+0000","last_undegraded":"2024-09-11T14:44:20.506789+0000","last_fullsized":"2024-09-11T14:44:20.506789+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":908,"log_dups_size":0,"ondisk_log_size":908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39059456,"num_objects":596,"num_object_clones":0,"num_object_copies":1788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":908,"num_write_kb":48128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.086961+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:42:13.086961+0000","last_peered":"2024-09-11T14:42:13.086961+0000","last_clean":"2024-09-11T14:42:13.086961+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:42:13.086961+0000","last_undegraded":"2024-09-11T14:42:13.086961+0000","last_fullsized":"2024-09-11T14:42:13.086961+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"27'1003","reported_seq":1039,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:19.602038+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:44:19.602038+0000","last_peered":"2024-09-11T14:44:19.602038+0000","last_clean":"2024-09-11T14:44:19.602038+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:44:19.602038+0000","last_undegraded":"2024-09-11T14:44:19.602038+0000","last_fullsized":"2024-09-11T14:44:19.602038+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1003,"log_dups_size":0,"ondisk_log_size":1003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42663936,"num_objects":651,"num_object_clones":0,"num_object_copies":1953,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1003,"num_write_kb":52928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111171+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:42:13.111171+0000","last_peered":"2024-09-11T14:42:13.111171+0000","last_clean":"2024-09-11T14:42:13.111171+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:42:13.111171+0000","last_undegraded":"2024-09-11T14:42:13.111171+0000","last_fullsized":"2024-09-11T14:42:13.111171+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"27'984","reported_seq":1020,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:20.695581+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:44:20.695581+0000","last_peered":"2024-09-11T14:44:20.695581+0000","last_clean":"2024-09-11T14:44:20.695581+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:44:20.695581+0000","last_undegraded":"2024-09-11T14:44:20.695581+0000","last_fullsized":"2024-09-11T14:44:20.695581+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":984,"log_dups_size":0,"ondisk_log_size":984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42205184,"num_objects":644,"num_object_clones":0,"num_object_copies":1932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":984,"num_write_kb":52096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.181385+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:42:13.181385+0000","last_peered":"2024-09-11T14:42:13.181385+0000","last_clean":"2024-09-11T14:42:13.181385+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:42:13.181385+0000","last_undegraded":"2024-09-11T14:42:13.181385+0000","last_fullsized":"2024-09-11T14:42:13.181385+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":682754072,"num_objects":10419,"num_object_clones":0,"num_object_copies":31257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15789,"num_write_kb":838593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":210053500928,"internally_reserved":0,"allocated":4157992960,"data_stored":4157992960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15789,"ondisk_log_size":15789,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":210053238784,"internally_reserved":0,"allocated":4158255104,"data_stored":4158255104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":52348014592,"internally_reserved":0,"allocated":1204858880,"data_stored":1204858880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673154,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":926988,"kb_used_data":926988,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51370740,"statfs":{"total":53552873472,"available":52603637760,"internally_reserved":0,"allocated":949235712,"data_stored":949235712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705858,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1185864,"kb_used_data":1185864,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51111864,"statfs":{"total":53552873472,"available":52338548736,"internally_reserved":0,"allocated":1214324736,"data_stored":1214324736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705858,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1039140,"kb_used_data":1039140,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51258588,"statfs":{"total":53552873472,"available":52488794112,"internally_reserved":0,"allocated":1064079360,"data_stored":1064079360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705858,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":936704,"kb_used_data":936704,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51361024,"statfs":{"total":53552873472,"available":52593688576,"internally_reserved":0,"allocated":959184896,"data_stored":959184896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":52348014592,"internally_reserved":0,"allocated":1204858880,"data_stored":1204858880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":52599328768,"internally_reserved":0,"allocated":953544704,"data_stored":953544704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":52495364096,"internally_reserved":0,"allocated":1057509376,"data_stored":1057509376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":52610531328,"internally_reserved":0,"allocated":942342144,"data_stored":942342144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":52348014592,"internally_reserved":0,"allocated":1204858880,"data_stored":1204858880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":52599328768,"internally_reserved":0,"allocated":953544704,"data_stored":953544704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":52495364096,"internally_reserved":0,"allocated":1057509376,"data_stored":1057509376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":52610531328,"internally_reserved":0,"allocated":942342144,"data_stored":942342144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":52348276736,"internally_reserved":0,"allocated":1204596736,"data_stored":1204596736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:44:27.082 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:44:27.253 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:27.248+0000 7faee50d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:27.260 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:27.259+0000 7faee50d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:29.788 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:44:29.931 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:29.927+0000 7f5064391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:29.936 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:29.935+0000 7f5064391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:30.867 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:44:30.867 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:44:30.867 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-11T14:44:30.867 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-11T14:44:31.023 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:31.013+0000 7f707cebd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:31.031 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:31.030+0000 7f707cebd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:31.058 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:44:31.058 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:44:31.058 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:44:31.058 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:44:31.059 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:44:31.059 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:44:31.059 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:44:31.059 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:44:31.059 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:44:31.059 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:44:31.059 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:44:31.059 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:44:31.063 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:44:31.063 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:44:31.063 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-11T14:44:31.243 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:31.231+0000 7f3a56791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:31.250 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:31.249+0000 7f3a56791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:31.327 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:44:31.327 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:44:31.327 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:44:31.327 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:44:31.327 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:44:31.327 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:44:31.327 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:44:31.327 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:44:31.327 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:44:31.327 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:44:31.328 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:44:31.328 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:44:31.330 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:44:31.330 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:44:31.331 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-11T14:44:31.480 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:31.468+0000 7f29374a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:31.487 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:31.486+0000 7f29374a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:31.618 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:44:31.619 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:44:31.619 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:44:31.619 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:44:31.619 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:44:31.619 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:44:31.619 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:44:31.619 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:44:31.619 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:44:31.619 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:44:31.619 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:44:31.619 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:44:31.624 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:44:31.624 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:44:31.625 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-11T14:44:31.808 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:31.799+0000 7f0a25f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:31.813 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:31.812+0000 7f0a25f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:31.899 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:44:31.899 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:44:31.900 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:44:31.900 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:44:31.900 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:44:31.900 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:44:31.900 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:44:31.900 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:44:31.900 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:44:31.900 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:44:31.900 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:44:31.900 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:44:31.903 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:44:31.904 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:44:32.236 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:44:32.394 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:32.390+0000 7f8549591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:32.399 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:32.398+0000 7f8549591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:34.864 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:44:34.997 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:34.993+0000 7f72b7afd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:35.001 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:35.000+0000 7f72b7afd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:36.905 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:44:36.905 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:44:36.905 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-11T14:44:36.905 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-11T14:44:37.035 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:37.027+0000 7fd5a98d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:37.039 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:37.037+0000 7fd5a98d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:37.053 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:44:37.053 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:44:37.054 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:44:37.054 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:44:37.054 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:44:37.054 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:44:37.054 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:44:37.054 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:44:37.054 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:44:37.054 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:44:37.054 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:44:37.054 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:44:37.059 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:44:37.059 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:44:37.060 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-11T14:44:37.173 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:44:37.186 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:37.176+0000 7f236af91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:37.191 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:37.190+0000 7f236af91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:37.252 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:44:37.252 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:44:37.252 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:44:37.252 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:44:37.252 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:44:37.252 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:44:37.252 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:44:37.252 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:44:37.252 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:44:37.252 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:44:37.252 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:44:37.253 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:44:37.256 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:44:37.256 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:44:37.257 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-11T14:44:37.328 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:37.323+0000 7f9c61d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:37.364 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:37.334+0000 7f9c61d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:37.436 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:37.428+0000 7fba6a311640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:37.440 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:37.439+0000 7fba6a311640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:37.484 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:44:37.485 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:44:37.485 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:44:37.485 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:44:37.485 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:44:37.485 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:44:37.485 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:44:37.485 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:44:37.485 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:44:37.485 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:44:37.485 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:44:37.485 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:44:37.488 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:44:37.489 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:44:37.489 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-11T14:44:37.699 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:37.691+0000 7fa9edf91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:37.707 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:37.705+0000 7fa9edf91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:37.824 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:44:37.824 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:44:37.824 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:44:37.824 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:44:37.824 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:44:37.824 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:44:37.824 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:44:37.824 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:44:37.824 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:44:37.824 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:44:37.825 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:44:37.825 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:44:37.829 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:44:37.829 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:44:39.599 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:44:39.733 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:39.730+0000 7f39c3991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:39.738 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:39.737+0000 7f39c3991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:42.004 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:44:42.146 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:42.143+0000 7f1bc7711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:42.152 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:42.151+0000 7f1bc7711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:42.829 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:44:42.830 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:44:42.830 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-11T14:44:42.830 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-11T14:44:42.986 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:42.978+0000 7f876daac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:42.992 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:42.990+0000 7f876daac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:43.058 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:44:43.059 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:44:43.059 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:44:43.059 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:44:43.059 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:44:43.059 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:44:43.059 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:44:43.059 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:44:43.059 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:44:43.060 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:44:43.060 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:44:43.060 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:44:43.063 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:44:43.063 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:44:43.063 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-11T14:44:43.235 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:43.222+0000 7f401f591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:43.244 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:43.242+0000 7f401f591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:43.361 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:44:43.362 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:44:43.362 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:44:43.362 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:44:43.362 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:44:43.362 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:44:43.362 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:44:43.362 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:44:43.362 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:44:43.362 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:44:43.362 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:44:43.362 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:44:43.366 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:44:43.366 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:44:43.366 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-11T14:44:43.553 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:43.541+0000 7feba5b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:43.560 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:43.559+0000 7feba5b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:43.589 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:44:43.589 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:44:43.589 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:44:43.589 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:44:43.589 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:44:43.589 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:44:43.589 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:44:43.589 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:44:43.589 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:44:43.589 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:44:43.589 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:44:43.590 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:44:43.593 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:44:43.593 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:44:43.594 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-11T14:44:43.761 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:43.752+0000 7fc939b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:43.765 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:43.764+0000 7fc939b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:43.954 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:44:43.954 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:44:43.954 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:44:43.954 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:44:43.954 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:44:43.954 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:44:43.955 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:44:43.955 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:44:43.955 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:44:43.955 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:44:43.955 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:44:43.955 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:44:43.957 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:44:43.958 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:44:44.323 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:44:44.478 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:44.474+0000 7f7be7d51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:44.486 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:44.484+0000 7f7be7d51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:47.161 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:44:47.321 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:47.317+0000 7f064cf91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:47.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:47.326+0000 7f064cf91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:48.959 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:44:48.959 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:44:48.959 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:44:49.097 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:49.090+0000 7fbb3d26c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:49.102 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:49.101+0000 7fbb3d26c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:49.421 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:44:49.421 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:44:49.438 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2024-09-11T14:44:48.754710+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":534577739,"num_objects":8153,"num_object_clones":0,"num_object_copies":24459,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":18140,"num_write_kb":839701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18114,"ondisk_log_size":18114,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":3220092,"kb_used_data":3220092,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":205970820,"statfs":{"total":214211493888,"available":210914119680,"internally_reserved":0,"allocated":3297374208,"data_stored":3297374208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-54788096,"num_objects":-836,"num_object_clones":0,"num_object_copies":-2508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":836,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002680"},"pg_stats":[{"pgid":"3.f","version":"27'1090","reported_seq":1126,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:44.205715+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:44:44.205715+0000","last_peered":"2024-09-11T14:44:44.205715+0000","last_clean":"2024-09-11T14:44:44.205715+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:44:44.205715+0000","last_undegraded":"2024-09-11T14:44:44.205715+0000","last_fullsized":"2024-09-11T14:44:44.205715+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1090,"log_dups_size":0,"ondisk_log_size":1090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34603008,"num_objects":528,"num_object_clones":0,"num_object_copies":1584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1090,"num_write_kb":51776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"27'1251","reported_seq":1287,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:45.437079+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:44:45.437079+0000","last_peered":"2024-09-11T14:44:45.437079+0000","last_clean":"2024-09-11T14:44:45.437079+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:44:45.437079+0000","last_undegraded":"2024-09-11T14:44:45.437079+0000","last_fullsized":"2024-09-11T14:44:45.437079+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1251,"log_dups_size":0,"ondisk_log_size":1251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34668544,"num_objects":529,"num_object_clones":0,"num_object_copies":1587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1251,"num_write_kb":56960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"27'1127","reported_seq":1164,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:44.576745+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:44:44.576745+0000","last_peered":"2024-09-11T14:44:44.576745+0000","last_clean":"2024-09-11T14:44:44.576745+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:44:44.576745+0000","last_undegraded":"2024-09-11T14:44:44.576745+0000","last_fullsized":"2024-09-11T14:44:44.576745+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1127,"log_dups_size":0,"ondisk_log_size":1127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35389464,"num_objects":541,"num_object_clones":0,"num_object_copies":1623,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1127,"num_write_kb":53313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"27'1118","reported_seq":1154,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:44.192779+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:44:44.192779+0000","last_peered":"2024-09-11T14:44:44.192779+0000","last_clean":"2024-09-11T14:44:44.192779+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:44:44.192779+0000","last_undegraded":"2024-09-11T14:44:44.192779+0000","last_fullsized":"2024-09-11T14:44:44.192779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1118,"log_dups_size":0,"ondisk_log_size":1118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30277632,"num_objects":462,"num_object_clones":0,"num_object_copies":1386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1118,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"27'1106","reported_seq":1142,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:44.597098+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:44:44.597098+0000","last_peered":"2024-09-11T14:44:44.597098+0000","last_clean":"2024-09-11T14:44:44.597098+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:44:44.597098+0000","last_undegraded":"2024-09-11T14:44:44.597098+0000","last_fullsized":"2024-09-11T14:44:44.597098+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1106,"log_dups_size":0,"ondisk_log_size":1106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32636928,"num_objects":498,"num_object_clones":0,"num_object_copies":1494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1106,"num_write_kb":51328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"27'1228","reported_seq":1264,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:44.618277+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:44:44.618277+0000","last_peered":"2024-09-11T14:44:44.618277+0000","last_clean":"2024-09-11T14:44:44.618277+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:44:44.618277+0000","last_undegraded":"2024-09-11T14:44:44.618277+0000","last_fullsized":"2024-09-11T14:44:44.618277+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1228,"log_dups_size":0,"ondisk_log_size":1228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35389440,"num_objects":540,"num_object_clones":0,"num_object_copies":1620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1228,"num_write_kb":56576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"27'1144","reported_seq":1180,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:44.738238+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:44:44.738238+0000","last_peered":"2024-09-11T14:44:44.738238+0000","last_clean":"2024-09-11T14:44:44.738238+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:44:44.738238+0000","last_undegraded":"2024-09-11T14:44:44.738238+0000","last_fullsized":"2024-09-11T14:44:44.738238+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1144,"log_dups_size":0,"ondisk_log_size":1144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34734080,"num_objects":530,"num_object_clones":0,"num_object_copies":1590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1144,"num_write_kb":53568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"27'1074","reported_seq":1110,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:44.751133+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:44:44.751133+0000","last_peered":"2024-09-11T14:44:44.751133+0000","last_clean":"2024-09-11T14:44:44.751133+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:44:44.751133+0000","last_undegraded":"2024-09-11T14:44:44.751133+0000","last_fullsized":"2024-09-11T14:44:44.751133+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34078720,"num_objects":520,"num_object_clones":0,"num_object_copies":1560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1074,"num_write_kb":51008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":121,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084543+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:42:13.084543+0000","last_peered":"2024-09-11T14:42:13.084543+0000","last_clean":"2024-09-11T14:42:13.084543+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:42:13.084543+0000","last_undegraded":"2024-09-11T14:42:13.084543+0000","last_fullsized":"2024-09-11T14:42:13.084543+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":50,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.184283+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:42:13.184283+0000","last_peered":"2024-09-11T14:42:13.184283+0000","last_clean":"2024-09-11T14:42:13.184283+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:42:13.184283+0000","last_undegraded":"2024-09-11T14:42:13.184283+0000","last_fullsized":"2024-09-11T14:42:13.184283+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"27'1119","reported_seq":1155,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:44.563636+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:44:44.563636+0000","last_peered":"2024-09-11T14:44:44.563636+0000","last_clean":"2024-09-11T14:44:44.563636+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:44:44.563636+0000","last_undegraded":"2024-09-11T14:44:44.563636+0000","last_fullsized":"2024-09-11T14:44:44.563636+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1119,"log_dups_size":0,"ondisk_log_size":1119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32964608,"num_objects":503,"num_object_clones":0,"num_object_copies":1509,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1119,"num_write_kb":51904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084333+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:42:13.084333+0000","last_peered":"2024-09-11T14:42:13.084333+0000","last_clean":"2024-09-11T14:42:13.084333+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:42:13.084333+0000","last_undegraded":"2024-09-11T14:42:13.084333+0000","last_fullsized":"2024-09-11T14:42:13.084333+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"27'1098","reported_seq":1134,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:44.593480+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:44:44.593480+0000","last_peered":"2024-09-11T14:44:44.593480+0000","last_clean":"2024-09-11T14:44:44.593480+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:44:44.593480+0000","last_undegraded":"2024-09-11T14:44:44.593480+0000","last_fullsized":"2024-09-11T14:44:44.593480+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1098,"log_dups_size":0,"ondisk_log_size":1098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34865152,"num_objects":532,"num_object_clones":0,"num_object_copies":1596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1098,"num_write_kb":52160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.179633+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:42:13.179633+0000","last_peered":"2024-09-11T14:42:13.179633+0000","last_clean":"2024-09-11T14:42:13.179633+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:42:13.179633+0000","last_undegraded":"2024-09-11T14:42:13.179633+0000","last_fullsized":"2024-09-11T14:42:13.179633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"27'1132","reported_seq":1168,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:44.365439+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:44:44.365439+0000","last_peered":"2024-09-11T14:44:44.365439+0000","last_clean":"2024-09-11T14:44:44.365439+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:44:44.365439+0000","last_undegraded":"2024-09-11T14:44:44.365439+0000","last_fullsized":"2024-09-11T14:44:44.365439+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1132,"log_dups_size":0,"ondisk_log_size":1132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32374784,"num_objects":494,"num_object_clones":0,"num_object_copies":1482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1132,"num_write_kb":52032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":51,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.082423+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:42:13.082423+0000","last_peered":"2024-09-11T14:42:13.082423+0000","last_clean":"2024-09-11T14:42:13.082423+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:42:13.082423+0000","last_undegraded":"2024-09-11T14:42:13.082423+0000","last_fullsized":"2024-09-11T14:42:13.082423+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"27'1108","reported_seq":1144,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:45.725678+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:44:45.725678+0000","last_peered":"2024-09-11T14:44:45.725678+0000","last_clean":"2024-09-11T14:44:45.725678+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:44:45.725678+0000","last_undegraded":"2024-09-11T14:44:45.725678+0000","last_fullsized":"2024-09-11T14:44:45.725678+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1108,"log_dups_size":0,"ondisk_log_size":1108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32112640,"num_objects":490,"num_object_clones":0,"num_object_copies":1470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1108,"num_write_kb":51136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"27'1125","reported_seq":1161,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:44.731363+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:44:44.731363+0000","last_peered":"2024-09-11T14:44:44.731363+0000","last_clean":"2024-09-11T14:44:44.731363+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:44:44.731363+0000","last_undegraded":"2024-09-11T14:44:44.731363+0000","last_fullsized":"2024-09-11T14:44:44.731363+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1125,"log_dups_size":0,"ondisk_log_size":1125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35061760,"num_objects":535,"num_object_clones":0,"num_object_copies":1605,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1125,"num_write_kb":53120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111507+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:42:13.111507+0000","last_peered":"2024-09-11T14:42:13.111507+0000","last_clean":"2024-09-11T14:42:13.111507+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:42:13.111507+0000","last_undegraded":"2024-09-11T14:42:13.111507+0000","last_fullsized":"2024-09-11T14:42:13.111507+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"27'1058","reported_seq":1094,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:45.151412+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:44:45.151412+0000","last_peered":"2024-09-11T14:44:45.151412+0000","last_clean":"2024-09-11T14:44:45.151412+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:44:45.151412+0000","last_undegraded":"2024-09-11T14:44:45.151412+0000","last_fullsized":"2024-09-11T14:44:45.151412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1058,"log_dups_size":0,"ondisk_log_size":1058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29229056,"num_objects":446,"num_object_clones":0,"num_object_copies":1338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1058,"num_write_kb":48128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.086961+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:42:13.086961+0000","last_peered":"2024-09-11T14:42:13.086961+0000","last_clean":"2024-09-11T14:42:13.086961+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:42:13.086961+0000","last_undegraded":"2024-09-11T14:42:13.086961+0000","last_fullsized":"2024-09-11T14:42:13.086961+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"27'1147","reported_seq":1183,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:44.621318+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:44:44.621318+0000","last_peered":"2024-09-11T14:44:44.621318+0000","last_clean":"2024-09-11T14:44:44.621318+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:44:44.621318+0000","last_undegraded":"2024-09-11T14:44:44.621318+0000","last_fullsized":"2024-09-11T14:44:44.621318+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1147,"log_dups_size":0,"ondisk_log_size":1147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33226752,"num_objects":507,"num_object_clones":0,"num_object_copies":1521,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1147,"num_write_kb":52928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111171+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:42:13.111171+0000","last_peered":"2024-09-11T14:42:13.111171+0000","last_clean":"2024-09-11T14:42:13.111171+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:42:13.111171+0000","last_undegraded":"2024-09-11T14:42:13.111171+0000","last_fullsized":"2024-09-11T14:42:13.111171+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"27'1134","reported_seq":1170,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:45.566382+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:44:45.566382+0000","last_peered":"2024-09-11T14:44:45.566382+0000","last_clean":"2024-09-11T14:44:45.566382+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:44:45.566382+0000","last_undegraded":"2024-09-11T14:44:45.566382+0000","last_fullsized":"2024-09-11T14:44:45.566382+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1134,"log_dups_size":0,"ondisk_log_size":1134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32374784,"num_objects":494,"num_object_clones":0,"num_object_copies":1482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1134,"num_write_kb":52096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.181385+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:42:13.181385+0000","last_peered":"2024-09-11T14:42:13.181385+0000","last_clean":"2024-09-11T14:42:13.181385+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:42:13.181385+0000","last_undegraded":"2024-09-11T14:42:13.181385+0000","last_fullsized":"2024-09-11T14:42:13.181385+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":533987352,"num_objects":8149,"num_object_clones":0,"num_object_copies":24447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18059,"num_write_kb":838593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":210944278528,"internally_reserved":0,"allocated":3267215360,"data_stored":3267215360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18059,"ondisk_log_size":18059,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":210943754240,"internally_reserved":0,"allocated":3267739648,"data_stored":3267739648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":52603510784,"internally_reserved":0,"allocated":949362688,"data_stored":949362688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673179,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":724072,"kb_used_data":724072,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51573656,"statfs":{"total":53552873472,"available":52811423744,"internally_reserved":0,"allocated":741449728,"data_stored":741449728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":935156,"kb_used_data":935156,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51362572,"statfs":{"total":53552873472,"available":52595273728,"internally_reserved":0,"allocated":957599744,"data_stored":957599744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":825032,"kb_used_data":825032,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51472696,"statfs":{"total":53552873472,"available":52708040704,"internally_reserved":0,"allocated":844832768,"data_stored":844832768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":735832,"kb_used_data":735832,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51561896,"statfs":{"total":53552873472,"available":52799381504,"internally_reserved":0,"allocated":753491968,"data_stored":753491968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":52603510784,"internally_reserved":0,"allocated":949362688,"data_stored":949362688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":52806074368,"internally_reserved":0,"allocated":746799104,"data_stored":746799104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":52716388352,"internally_reserved":0,"allocated":836485120,"data_stored":836485120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":52817649664,"internally_reserved":0,"allocated":735223808,"data_stored":735223808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":52603641856,"internally_reserved":0,"allocated":949231616,"data_stored":949231616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":52806074368,"internally_reserved":0,"allocated":746799104,"data_stored":746799104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":52716388352,"internally_reserved":0,"allocated":836485120,"data_stored":836485120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":52818042880,"internally_reserved":0,"allocated":734830592,"data_stored":734830592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":52603772928,"internally_reserved":0,"allocated":949100544,"data_stored":949100544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:44:49.798 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:44:49.974 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:49.971+0000 7ff62e191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:49.979 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:49.977+0000 7ff62e191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:52.398 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:44:52.556 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:52.551+0000 7f606bf91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:52.560 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:52.558+0000 7f606bf91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:54.440 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:44:54.440 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:44:54.440 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-11T14:44:54.440 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-11T14:44:54.618 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:54.610+0000 7f19cc4fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:54.624 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:54.623+0000 7f19cc4fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:54.724 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:44:54.734 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:44:54.734 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:44:54.734 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:44:54.734 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:44:54.734 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:44:54.734 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:44:54.734 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:44:54.735 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:44:54.735 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:44:54.735 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:44:54.735 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:44:54.735 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:44:54.738 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:44:54.738 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:44:54.739 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-11T14:44:54.874 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:54.870+0000 7fe0e5893640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:54.879 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:54.878+0000 7fe0e5893640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:54.944 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:54.936+0000 7fd161591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:54.950 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:54.949+0000 7fd161591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:55.160 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:44:55.160 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:44:55.160 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:44:55.161 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:44:55.161 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:44:55.161 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:44:55.161 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:44:55.161 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:44:55.161 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:44:55.161 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:44:55.161 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:44:55.162 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:44:55.165 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:44:55.165 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:44:55.166 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-11T14:44:55.345 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:55.332+0000 7f54ecb11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:55.356 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:55.354+0000 7f54ecb11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:55.420 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:44:55.420 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:44:55.420 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:44:55.420 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:44:55.420 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:44:55.420 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:44:55.420 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:44:55.420 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:44:55.420 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:44:55.421 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:44:55.421 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:44:55.421 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:44:55.424 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:44:55.425 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:44:55.425 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-11T14:44:55.633 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:55.620+0000 7f86c3311640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:55.640 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:55.639+0000 7f86c3311640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:55.777 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:44:55.777 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:44:55.777 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:44:55.778 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:44:55.778 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:44:55.778 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:44:55.778 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:44:55.778 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:44:55.778 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:44:55.778 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:44:55.778 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:44:55.778 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:44:55.781 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:44:55.782 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:44:57.411 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:44:57.575 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:57.569+0000 7f9d0af91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:44:57.581 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:44:57.580+0000 7f9d0af91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:00.008 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:45:00.168 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:00.164+0000 7f5d42391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:00.174 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:00.172+0000 7f5d42391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:00.783 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:45:00.783 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:45:00.783 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:45:00.953 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:00.946+0000 7fd743591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:00.960 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:00.959+0000 7fd743591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:01.331 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:45:01.332 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:45:01.347 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-09-11T14:45:00.757634+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":454296139,"num_objects":6928,"num_object_clones":0,"num_object_copies":20784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":19365,"num_write_kb":839701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19339,"ondisk_log_size":19339,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":2735436,"kb_used_data":2735436,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":206455476,"statfs":{"total":214211493888,"available":211410407424,"internally_reserved":0,"allocated":2801086464,"data_stored":2801086464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-80281600,"num_objects":-1225,"num_object_clones":0,"num_object_copies":-3675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1225,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002923"},"pg_stats":[{"pgid":"3.f","version":"27'1177","reported_seq":1213,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:59.846095+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:44:59.846095+0000","last_peered":"2024-09-11T14:44:59.846095+0000","last_clean":"2024-09-11T14:44:59.846095+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:44:59.846095+0000","last_undegraded":"2024-09-11T14:44:59.846095+0000","last_fullsized":"2024-09-11T14:44:59.846095+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1177,"log_dups_size":0,"ondisk_log_size":1177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28901376,"num_objects":441,"num_object_clones":0,"num_object_copies":1323,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1177,"num_write_kb":51776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"27'1299","reported_seq":1335,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:55.711756+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:44:55.711756+0000","last_peered":"2024-09-11T14:44:55.711756+0000","last_clean":"2024-09-11T14:44:55.711756+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:44:55.711756+0000","last_undegraded":"2024-09-11T14:44:55.711756+0000","last_fullsized":"2024-09-11T14:44:55.711756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1299,"log_dups_size":0,"ondisk_log_size":1299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31522816,"num_objects":481,"num_object_clones":0,"num_object_copies":1443,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1299,"num_write_kb":56960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"27'1211","reported_seq":1248,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:59.445538+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:44:59.445538+0000","last_peered":"2024-09-11T14:44:59.445538+0000","last_clean":"2024-09-11T14:44:59.445538+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:44:59.445538+0000","last_undegraded":"2024-09-11T14:44:59.445538+0000","last_fullsized":"2024-09-11T14:44:59.445538+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1211,"log_dups_size":0,"ondisk_log_size":1211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29884440,"num_objects":457,"num_object_clones":0,"num_object_copies":1371,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1211,"num_write_kb":53313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"27'1193","reported_seq":1229,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:59.563672+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:44:59.563672+0000","last_peered":"2024-09-11T14:44:59.563672+0000","last_clean":"2024-09-11T14:44:59.563672+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:44:59.563672+0000","last_undegraded":"2024-09-11T14:44:59.563672+0000","last_fullsized":"2024-09-11T14:44:59.563672+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1193,"log_dups_size":0,"ondisk_log_size":1193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25362432,"num_objects":387,"num_object_clones":0,"num_object_copies":1161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1193,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"27'1200","reported_seq":1236,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:59.455056+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:44:59.455056+0000","last_peered":"2024-09-11T14:44:59.455056+0000","last_clean":"2024-09-11T14:44:59.455056+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:44:59.455056+0000","last_undegraded":"2024-09-11T14:44:59.455056+0000","last_fullsized":"2024-09-11T14:44:59.455056+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1200,"log_dups_size":0,"ondisk_log_size":1200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26476544,"num_objects":404,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1200,"num_write_kb":51328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"27'1313","reported_seq":1349,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:58.537892+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:44:58.537892+0000","last_peered":"2024-09-11T14:44:58.537892+0000","last_clean":"2024-09-11T14:44:58.537892+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:44:58.537892+0000","last_undegraded":"2024-09-11T14:44:58.537892+0000","last_fullsized":"2024-09-11T14:44:58.537892+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1313,"log_dups_size":0,"ondisk_log_size":1313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29818880,"num_objects":455,"num_object_clones":0,"num_object_copies":1365,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1313,"num_write_kb":56576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"27'1238","reported_seq":1274,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:59.832451+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:44:59.832451+0000","last_peered":"2024-09-11T14:44:59.832451+0000","last_clean":"2024-09-11T14:44:59.832451+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:44:59.832451+0000","last_undegraded":"2024-09-11T14:44:59.832451+0000","last_fullsized":"2024-09-11T14:44:59.832451+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1238,"log_dups_size":0,"ondisk_log_size":1238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28573696,"num_objects":436,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1238,"num_write_kb":53568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"27'1156","reported_seq":1192,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:59.054304+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:44:59.054304+0000","last_peered":"2024-09-11T14:44:59.054304+0000","last_clean":"2024-09-11T14:44:59.054304+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:44:59.054304+0000","last_undegraded":"2024-09-11T14:44:59.054304+0000","last_fullsized":"2024-09-11T14:44:59.054304+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1156,"log_dups_size":0,"ondisk_log_size":1156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28704768,"num_objects":438,"num_object_clones":0,"num_object_copies":1314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1156,"num_write_kb":51008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":121,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084543+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:42:13.084543+0000","last_peered":"2024-09-11T14:42:13.084543+0000","last_clean":"2024-09-11T14:42:13.084543+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:42:13.084543+0000","last_undegraded":"2024-09-11T14:42:13.084543+0000","last_fullsized":"2024-09-11T14:42:13.084543+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":50,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.184283+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:42:13.184283+0000","last_peered":"2024-09-11T14:42:13.184283+0000","last_clean":"2024-09-11T14:42:13.184283+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:42:13.184283+0000","last_undegraded":"2024-09-11T14:42:13.184283+0000","last_fullsized":"2024-09-11T14:42:13.184283+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"27'1203","reported_seq":1239,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:59.489629+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:44:59.489629+0000","last_peered":"2024-09-11T14:44:59.489629+0000","last_clean":"2024-09-11T14:44:59.489629+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:44:59.489629+0000","last_undegraded":"2024-09-11T14:44:59.489629+0000","last_fullsized":"2024-09-11T14:44:59.489629+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1203,"log_dups_size":0,"ondisk_log_size":1203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27459584,"num_objects":419,"num_object_clones":0,"num_object_copies":1257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1203,"num_write_kb":51904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084333+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:42:13.084333+0000","last_peered":"2024-09-11T14:42:13.084333+0000","last_clean":"2024-09-11T14:42:13.084333+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:42:13.084333+0000","last_undegraded":"2024-09-11T14:42:13.084333+0000","last_fullsized":"2024-09-11T14:42:13.084333+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"27'1185","reported_seq":1221,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:59.196101+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:44:59.196101+0000","last_peered":"2024-09-11T14:44:59.196101+0000","last_clean":"2024-09-11T14:44:59.196101+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:44:59.196101+0000","last_undegraded":"2024-09-11T14:44:59.196101+0000","last_fullsized":"2024-09-11T14:44:59.196101+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1185,"log_dups_size":0,"ondisk_log_size":1185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29163520,"num_objects":445,"num_object_clones":0,"num_object_copies":1335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1185,"num_write_kb":52160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.179633+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:42:13.179633+0000","last_peered":"2024-09-11T14:42:13.179633+0000","last_clean":"2024-09-11T14:42:13.179633+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:42:13.179633+0000","last_undegraded":"2024-09-11T14:42:13.179633+0000","last_fullsized":"2024-09-11T14:42:13.179633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"27'1204","reported_seq":1240,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:59.433101+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:44:59.433101+0000","last_peered":"2024-09-11T14:44:59.433101+0000","last_clean":"2024-09-11T14:44:59.433101+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:44:59.433101+0000","last_undegraded":"2024-09-11T14:44:59.433101+0000","last_fullsized":"2024-09-11T14:44:59.433101+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1204,"log_dups_size":0,"ondisk_log_size":1204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27656192,"num_objects":422,"num_object_clones":0,"num_object_copies":1266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1204,"num_write_kb":52032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":51,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.082423+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:42:13.082423+0000","last_peered":"2024-09-11T14:42:13.082423+0000","last_clean":"2024-09-11T14:42:13.082423+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:42:13.082423+0000","last_undegraded":"2024-09-11T14:42:13.082423+0000","last_fullsized":"2024-09-11T14:42:13.082423+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"27'1157","reported_seq":1193,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:55.645887+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:44:55.645887+0000","last_peered":"2024-09-11T14:44:55.645887+0000","last_clean":"2024-09-11T14:44:55.645887+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:44:55.645887+0000","last_undegraded":"2024-09-11T14:44:55.645887+0000","last_fullsized":"2024-09-11T14:44:55.645887+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1157,"log_dups_size":0,"ondisk_log_size":1157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28901376,"num_objects":441,"num_object_clones":0,"num_object_copies":1323,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1157,"num_write_kb":51136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"27'1221","reported_seq":1257,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:59.314743+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:44:59.314743+0000","last_peered":"2024-09-11T14:44:59.314743+0000","last_clean":"2024-09-11T14:44:59.314743+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:44:59.314743+0000","last_undegraded":"2024-09-11T14:44:59.314743+0000","last_fullsized":"2024-09-11T14:44:59.314743+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1221,"log_dups_size":0,"ondisk_log_size":1221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28770304,"num_objects":439,"num_object_clones":0,"num_object_copies":1317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1221,"num_write_kb":53120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111507+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:42:13.111507+0000","last_peered":"2024-09-11T14:42:13.111507+0000","last_clean":"2024-09-11T14:42:13.111507+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:42:13.111507+0000","last_undegraded":"2024-09-11T14:42:13.111507+0000","last_fullsized":"2024-09-11T14:42:13.111507+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"27'1110","reported_seq":1146,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:55.091231+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:44:55.091231+0000","last_peered":"2024-09-11T14:44:55.091231+0000","last_clean":"2024-09-11T14:44:55.091231+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:44:55.091231+0000","last_undegraded":"2024-09-11T14:44:55.091231+0000","last_fullsized":"2024-09-11T14:44:55.091231+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1110,"log_dups_size":0,"ondisk_log_size":1110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25821184,"num_objects":394,"num_object_clones":0,"num_object_copies":1182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1110,"num_write_kb":48128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.086961+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:42:13.086961+0000","last_peered":"2024-09-11T14:42:13.086961+0000","last_clean":"2024-09-11T14:42:13.086961+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:42:13.086961+0000","last_undegraded":"2024-09-11T14:42:13.086961+0000","last_fullsized":"2024-09-11T14:42:13.086961+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"27'1223","reported_seq":1259,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:59.659248+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:44:59.659248+0000","last_peered":"2024-09-11T14:44:59.659248+0000","last_clean":"2024-09-11T14:44:59.659248+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:44:59.659248+0000","last_undegraded":"2024-09-11T14:44:59.659248+0000","last_fullsized":"2024-09-11T14:44:59.659248+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1223,"log_dups_size":0,"ondisk_log_size":1223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28246016,"num_objects":431,"num_object_clones":0,"num_object_copies":1293,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1223,"num_write_kb":52928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111171+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:42:13.111171+0000","last_peered":"2024-09-11T14:42:13.111171+0000","last_clean":"2024-09-11T14:42:13.111171+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:42:13.111171+0000","last_undegraded":"2024-09-11T14:42:13.111171+0000","last_fullsized":"2024-09-11T14:42:13.111171+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"27'1194","reported_seq":1230,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:44:55.757309+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:44:55.757309+0000","last_peered":"2024-09-11T14:44:55.757309+0000","last_clean":"2024-09-11T14:44:55.757309+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:44:55.757309+0000","last_undegraded":"2024-09-11T14:44:55.757309+0000","last_fullsized":"2024-09-11T14:44:55.757309+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1194,"log_dups_size":0,"ondisk_log_size":1194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28442624,"num_objects":434,"num_object_clones":0,"num_object_copies":1302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1194,"num_write_kb":52096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.181385+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:42:13.181385+0000","last_peered":"2024-09-11T14:42:13.181385+0000","last_clean":"2024-09-11T14:42:13.181385+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:42:13.181385+0000","last_undegraded":"2024-09-11T14:42:13.181385+0000","last_fullsized":"2024-09-11T14:42:13.181385+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":453705752,"num_objects":6924,"num_object_clones":0,"num_object_copies":20772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":19284,"num_write_kb":838593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":211434409984,"internally_reserved":0,"allocated":2777083904,"data_stored":2777083904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19284,"ondisk_log_size":19284,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":211434278912,"internally_reserved":0,"allocated":2777214976,"data_stored":2777214976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":52756885504,"internally_reserved":0,"allocated":795987968,"data_stored":795987968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673189,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":643900,"kb_used_data":643900,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51653828,"statfs":{"total":53552873472,"available":52893519872,"internally_reserved":0,"allocated":659353600,"data_stored":659353600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":784212,"kb_used_data":784212,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51513516,"statfs":{"total":53552873472,"available":52749840384,"internally_reserved":0,"allocated":803033088,"data_stored":803033088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":690784,"kb_used_data":690784,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51606944,"statfs":{"total":53552873472,"available":52845510656,"internally_reserved":0,"allocated":707362816,"data_stored":707362816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":616540,"kb_used_data":616540,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51681188,"statfs":{"total":53552873472,"available":52921536512,"internally_reserved":0,"allocated":631336960,"data_stored":631336960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":52756885504,"internally_reserved":0,"allocated":795987968,"data_stored":795987968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":52925689856,"internally_reserved":0,"allocated":627183616,"data_stored":627183616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":52850941952,"internally_reserved":0,"allocated":701931520,"data_stored":701931520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":52900630528,"internally_reserved":0,"allocated":652242944,"data_stored":652242944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":52757016576,"internally_reserved":0,"allocated":795856896,"data_stored":795856896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":52925820928,"internally_reserved":0,"allocated":627052544,"data_stored":627052544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":52850941952,"internally_reserved":0,"allocated":701931520,"data_stored":701931520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":52900630528,"internally_reserved":0,"allocated":652242944,"data_stored":652242944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":52757016576,"internally_reserved":0,"allocated":795856896,"data_stored":795856896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:45:02.611 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:45:02.781 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:02.777+0000 7efe6f151640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:02.787 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:02.785+0000 7efe6f151640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:05.434 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:45:05.585 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:05.582+0000 7fabd2711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:05.589 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:05.588+0000 7fabd2711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:06.349 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:45:06.350 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:45:06.350 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:45:06.529 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:06.510+0000 7f34add51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:06.537 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:06.536+0000 7f34add51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:06.863 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:45:06.864 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:45:06.889 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":141,"stamp":"2024-09-11T14:45:06.759076+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":417137227,"num_objects":6361,"num_object_clones":0,"num_object_copies":19083,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":19932,"num_write_kb":839701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19906,"ondisk_log_size":19906,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":2528752,"kb_used_data":2528752,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":206662160,"statfs":{"total":214211493888,"available":211622051840,"internally_reserved":0,"allocated":2589442048,"data_stored":2589442048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-86900736,"num_objects":-1326,"num_object_clones":0,"num_object_copies":-3978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1326,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002894"},"pg_stats":[{"pgid":"3.f","version":"27'1209","reported_seq":1245,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:04.636830+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:45:04.636830+0000","last_peered":"2024-09-11T14:45:04.636830+0000","last_clean":"2024-09-11T14:45:04.636830+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:45:04.636830+0000","last_undegraded":"2024-09-11T14:45:04.636830+0000","last_fullsized":"2024-09-11T14:45:04.636830+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1209,"log_dups_size":0,"ondisk_log_size":1209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26804224,"num_objects":409,"num_object_clones":0,"num_object_copies":1227,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1209,"num_write_kb":51776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"27'1367","reported_seq":1403,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:05.609602+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:45:05.609602+0000","last_peered":"2024-09-11T14:45:05.609602+0000","last_clean":"2024-09-11T14:45:05.609602+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:45:05.609602+0000","last_undegraded":"2024-09-11T14:45:05.609602+0000","last_fullsized":"2024-09-11T14:45:05.609602+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1367,"log_dups_size":0,"ondisk_log_size":1367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27066368,"num_objects":413,"num_object_clones":0,"num_object_copies":1239,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1367,"num_write_kb":56960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"27'1238","reported_seq":1275,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:04.756750+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:45:04.756750+0000","last_peered":"2024-09-11T14:45:04.756750+0000","last_clean":"2024-09-11T14:45:04.756750+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:45:04.756750+0000","last_undegraded":"2024-09-11T14:45:04.756750+0000","last_fullsized":"2024-09-11T14:45:04.756750+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1238,"log_dups_size":0,"ondisk_log_size":1238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28114968,"num_objects":430,"num_object_clones":0,"num_object_copies":1290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1238,"num_write_kb":53313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"27'1212","reported_seq":1248,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:04.048503+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:45:04.048503+0000","last_peered":"2024-09-11T14:45:04.048503+0000","last_clean":"2024-09-11T14:45:04.048503+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:45:04.048503+0000","last_undegraded":"2024-09-11T14:45:04.048503+0000","last_fullsized":"2024-09-11T14:45:04.048503+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1212,"log_dups_size":0,"ondisk_log_size":1212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24117248,"num_objects":368,"num_object_clones":0,"num_object_copies":1104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1212,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"27'1223","reported_seq":1259,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:04.505920+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:45:04.505920+0000","last_peered":"2024-09-11T14:45:04.505920+0000","last_clean":"2024-09-11T14:45:04.505920+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:45:04.505920+0000","last_undegraded":"2024-09-11T14:45:04.505920+0000","last_fullsized":"2024-09-11T14:45:04.505920+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1223,"log_dups_size":0,"ondisk_log_size":1223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24969216,"num_objects":381,"num_object_clones":0,"num_object_copies":1143,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1223,"num_write_kb":51328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"27'1341","reported_seq":1377,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:04.653894+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:45:04.653894+0000","last_peered":"2024-09-11T14:45:04.653894+0000","last_clean":"2024-09-11T14:45:04.653894+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:45:04.653894+0000","last_undegraded":"2024-09-11T14:45:04.653894+0000","last_fullsized":"2024-09-11T14:45:04.653894+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1341,"log_dups_size":0,"ondisk_log_size":1341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27983872,"num_objects":427,"num_object_clones":0,"num_object_copies":1281,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1341,"num_write_kb":56576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"27'1271","reported_seq":1307,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:04.682893+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:45:04.682893+0000","last_peered":"2024-09-11T14:45:04.682893+0000","last_clean":"2024-09-11T14:45:04.682893+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:45:04.682893+0000","last_undegraded":"2024-09-11T14:45:04.682893+0000","last_fullsized":"2024-09-11T14:45:04.682893+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1271,"log_dups_size":0,"ondisk_log_size":1271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26411008,"num_objects":403,"num_object_clones":0,"num_object_copies":1209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1271,"num_write_kb":53568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"27'1196","reported_seq":1232,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:04.802307+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:45:04.802307+0000","last_peered":"2024-09-11T14:45:04.802307+0000","last_clean":"2024-09-11T14:45:04.802307+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:45:04.802307+0000","last_undegraded":"2024-09-11T14:45:04.802307+0000","last_fullsized":"2024-09-11T14:45:04.802307+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1196,"log_dups_size":0,"ondisk_log_size":1196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26083328,"num_objects":398,"num_object_clones":0,"num_object_copies":1194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1196,"num_write_kb":51008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":121,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084543+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:42:13.084543+0000","last_peered":"2024-09-11T14:42:13.084543+0000","last_clean":"2024-09-11T14:42:13.084543+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:42:13.084543+0000","last_undegraded":"2024-09-11T14:42:13.084543+0000","last_fullsized":"2024-09-11T14:42:13.084543+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":50,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.184283+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:42:13.184283+0000","last_peered":"2024-09-11T14:42:13.184283+0000","last_clean":"2024-09-11T14:42:13.184283+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:42:13.184283+0000","last_undegraded":"2024-09-11T14:42:13.184283+0000","last_fullsized":"2024-09-11T14:42:13.184283+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"27'1226","reported_seq":1262,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:04.335868+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:45:04.335868+0000","last_peered":"2024-09-11T14:45:04.335868+0000","last_clean":"2024-09-11T14:45:04.335868+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:45:04.335868+0000","last_undegraded":"2024-09-11T14:45:04.335868+0000","last_fullsized":"2024-09-11T14:45:04.335868+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1226,"log_dups_size":0,"ondisk_log_size":1226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25952256,"num_objects":396,"num_object_clones":0,"num_object_copies":1188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1226,"num_write_kb":51904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084333+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:42:13.084333+0000","last_peered":"2024-09-11T14:42:13.084333+0000","last_clean":"2024-09-11T14:42:13.084333+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:42:13.084333+0000","last_undegraded":"2024-09-11T14:42:13.084333+0000","last_fullsized":"2024-09-11T14:42:13.084333+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"27'1214","reported_seq":1250,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:04.563359+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:45:04.563359+0000","last_peered":"2024-09-11T14:45:04.563359+0000","last_clean":"2024-09-11T14:45:04.563359+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:45:04.563359+0000","last_undegraded":"2024-09-11T14:45:04.563359+0000","last_fullsized":"2024-09-11T14:45:04.563359+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1214,"log_dups_size":0,"ondisk_log_size":1214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27262976,"num_objects":416,"num_object_clones":0,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1214,"num_write_kb":52160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.179633+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:42:13.179633+0000","last_peered":"2024-09-11T14:42:13.179633+0000","last_clean":"2024-09-11T14:42:13.179633+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:42:13.179633+0000","last_undegraded":"2024-09-11T14:42:13.179633+0000","last_fullsized":"2024-09-11T14:42:13.179633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"27'1223","reported_seq":1259,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:04.328295+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:45:04.328295+0000","last_peered":"2024-09-11T14:45:04.328295+0000","last_clean":"2024-09-11T14:45:04.328295+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:45:04.328295+0000","last_undegraded":"2024-09-11T14:45:04.328295+0000","last_fullsized":"2024-09-11T14:45:04.328295+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1223,"log_dups_size":0,"ondisk_log_size":1223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26411008,"num_objects":403,"num_object_clones":0,"num_object_copies":1209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1223,"num_write_kb":52032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":51,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.082423+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:42:13.082423+0000","last_peered":"2024-09-11T14:42:13.082423+0000","last_clean":"2024-09-11T14:42:13.082423+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:42:13.082423+0000","last_undegraded":"2024-09-11T14:42:13.082423+0000","last_fullsized":"2024-09-11T14:42:13.082423+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"27'1210","reported_seq":1246,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:05.849828+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:45:05.849828+0000","last_peered":"2024-09-11T14:45:05.849828+0000","last_clean":"2024-09-11T14:45:05.849828+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:45:05.849828+0000","last_undegraded":"2024-09-11T14:45:05.849828+0000","last_fullsized":"2024-09-11T14:45:05.849828+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1210,"log_dups_size":0,"ondisk_log_size":1210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25427968,"num_objects":388,"num_object_clones":0,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1210,"num_write_kb":51136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"27'1246","reported_seq":1282,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:04.738320+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:45:04.738320+0000","last_peered":"2024-09-11T14:45:04.738320+0000","last_clean":"2024-09-11T14:45:04.738320+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:45:04.738320+0000","last_undegraded":"2024-09-11T14:45:04.738320+0000","last_fullsized":"2024-09-11T14:45:04.738320+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1246,"log_dups_size":0,"ondisk_log_size":1246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27131904,"num_objects":414,"num_object_clones":0,"num_object_copies":1242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1246,"num_write_kb":53120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111507+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:42:13.111507+0000","last_peered":"2024-09-11T14:42:13.111507+0000","last_clean":"2024-09-11T14:42:13.111507+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:42:13.111507+0000","last_undegraded":"2024-09-11T14:42:13.111507+0000","last_fullsized":"2024-09-11T14:42:13.111507+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"27'1167","reported_seq":1203,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:05.305676+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:45:05.305676+0000","last_peered":"2024-09-11T14:45:05.305676+0000","last_clean":"2024-09-11T14:45:05.305676+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:45:05.305676+0000","last_undegraded":"2024-09-11T14:45:05.305676+0000","last_fullsized":"2024-09-11T14:45:05.305676+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1167,"log_dups_size":0,"ondisk_log_size":1167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22085632,"num_objects":337,"num_object_clones":0,"num_object_copies":1011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1167,"num_write_kb":48128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.086961+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:42:13.086961+0000","last_peered":"2024-09-11T14:42:13.086961+0000","last_clean":"2024-09-11T14:42:13.086961+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:42:13.086961+0000","last_undegraded":"2024-09-11T14:42:13.086961+0000","last_fullsized":"2024-09-11T14:42:13.086961+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"27'1256","reported_seq":1292,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:04.683762+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:45:04.683762+0000","last_peered":"2024-09-11T14:45:04.683762+0000","last_clean":"2024-09-11T14:45:04.683762+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:45:04.683762+0000","last_undegraded":"2024-09-11T14:45:04.683762+0000","last_fullsized":"2024-09-11T14:45:04.683762+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1256,"log_dups_size":0,"ondisk_log_size":1256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26083328,"num_objects":398,"num_object_clones":0,"num_object_copies":1194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1256,"num_write_kb":52928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111171+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:42:13.111171+0000","last_peered":"2024-09-11T14:42:13.111171+0000","last_clean":"2024-09-11T14:42:13.111171+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:42:13.111171+0000","last_undegraded":"2024-09-11T14:42:13.111171+0000","last_fullsized":"2024-09-11T14:42:13.111171+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"27'1252","reported_seq":1288,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:05.697534+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:45:05.697534+0000","last_peered":"2024-09-11T14:45:05.697534+0000","last_clean":"2024-09-11T14:45:05.697534+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:45:05.697534+0000","last_undegraded":"2024-09-11T14:45:05.697534+0000","last_fullsized":"2024-09-11T14:45:05.697534+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1252,"log_dups_size":0,"ondisk_log_size":1252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24641536,"num_objects":376,"num_object_clones":0,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1252,"num_write_kb":52096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.181385+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:42:13.181385+0000","last_peered":"2024-09-11T14:42:13.181385+0000","last_clean":"2024-09-11T14:42:13.181385+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:42:13.181385+0000","last_undegraded":"2024-09-11T14:42:13.181385+0000","last_fullsized":"2024-09-11T14:42:13.181385+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":416546840,"num_objects":6357,"num_object_clones":0,"num_object_copies":19071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":19851,"num_write_kb":838593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":211649527808,"internally_reserved":0,"allocated":2561966080,"data_stored":2561966080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19851,"ondisk_log_size":19851,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":211649265664,"internally_reserved":0,"allocated":2562228224,"data_stored":2562228224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":52808151040,"internally_reserved":0,"allocated":744722432,"data_stored":744722432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673199,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":565640,"kb_used_data":565640,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51732088,"statfs":{"total":53552873472,"available":52973658112,"internally_reserved":0,"allocated":579215360,"data_stored":579215360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705903,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":735360,"kb_used_data":735360,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51562368,"statfs":{"total":53552873472,"available":52799864832,"internally_reserved":0,"allocated":753008640,"data_stored":753008640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705903,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":649504,"kb_used_data":649504,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51648224,"statfs":{"total":53552873472,"available":52887781376,"internally_reserved":0,"allocated":665092096,"data_stored":665092096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705903,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":578248,"kb_used_data":578248,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51719480,"statfs":{"total":53552873472,"available":52960747520,"internally_reserved":0,"allocated":592125952,"data_stored":592125952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":52808151040,"internally_reserved":0,"allocated":744722432,"data_stored":744722432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":52967079936,"internally_reserved":0,"allocated":585793536,"data_stored":585793536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":52895473664,"internally_reserved":0,"allocated":657399808,"data_stored":657399808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":52978561024,"internally_reserved":0,"allocated":574312448,"data_stored":574312448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":52808151040,"internally_reserved":0,"allocated":744722432,"data_stored":744722432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":52967079936,"internally_reserved":0,"allocated":585793536,"data_stored":585793536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":52895735808,"internally_reserved":0,"allocated":657137664,"data_stored":657137664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":52978561024,"internally_reserved":0,"allocated":574312448,"data_stored":574312448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":52808151040,"internally_reserved":0,"allocated":744722432,"data_stored":744722432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:45:07.978 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:45:08.134 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:08.130+0000 7fc7f7551640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:08.138 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:08.137+0000 7fc7f7551640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:10.370 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:45:10.499 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:10.496+0000 7fa7fe46c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:10.504 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:10.503+0000 7fa7fe46c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:11.892 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:45:11.892 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:45:11.893 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:45:12.034 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:12.026+0000 7fd778cbd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:12.040 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:12.039+0000 7fd778cbd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:12.417 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:45:12.418 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:45:12.442 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":143,"stamp":"2024-09-11T14:45:10.760082+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":395575883,"num_objects":6032,"num_object_clones":0,"num_object_copies":18096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":20261,"num_write_kb":839701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20235,"ondisk_log_size":20235,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":2393952,"kb_used_data":2393952,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":206796960,"statfs":{"total":214211493888,"available":211760087040,"internally_reserved":0,"allocated":2451406848,"data_stored":2451406848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-80216064,"num_objects":-1224,"num_object_clones":0,"num_object_copies":-3672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1224,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002895"},"pg_stats":[{"pgid":"3.f","version":"27'1233","reported_seq":1269,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:09.731384+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:45:09.731384+0000","last_peered":"2024-09-11T14:45:09.731384+0000","last_clean":"2024-09-11T14:45:09.731384+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:45:09.731384+0000","last_undegraded":"2024-09-11T14:45:09.731384+0000","last_fullsized":"2024-09-11T14:45:09.731384+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1233,"log_dups_size":0,"ondisk_log_size":1233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25231360,"num_objects":385,"num_object_clones":0,"num_object_copies":1155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1233,"num_write_kb":51776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"27'1367","reported_seq":1403,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:05.609602+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:45:05.609602+0000","last_peered":"2024-09-11T14:45:05.609602+0000","last_clean":"2024-09-11T14:45:05.609602+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:45:05.609602+0000","last_undegraded":"2024-09-11T14:45:05.609602+0000","last_fullsized":"2024-09-11T14:45:05.609602+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1367,"log_dups_size":0,"ondisk_log_size":1367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27066368,"num_objects":413,"num_object_clones":0,"num_object_copies":1239,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1367,"num_write_kb":56960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"27'1269","reported_seq":1306,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:09.183791+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:45:09.183791+0000","last_peered":"2024-09-11T14:45:09.183791+0000","last_clean":"2024-09-11T14:45:09.183791+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:45:09.183791+0000","last_undegraded":"2024-09-11T14:45:09.183791+0000","last_fullsized":"2024-09-11T14:45:09.183791+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1269,"log_dups_size":0,"ondisk_log_size":1269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26083352,"num_objects":399,"num_object_clones":0,"num_object_copies":1197,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1269,"num_write_kb":53313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"27'1238","reported_seq":1274,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:09.739117+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:45:09.739117+0000","last_peered":"2024-09-11T14:45:09.739117+0000","last_clean":"2024-09-11T14:45:09.739117+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:45:09.739117+0000","last_undegraded":"2024-09-11T14:45:09.739117+0000","last_fullsized":"2024-09-11T14:45:09.739117+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1238,"log_dups_size":0,"ondisk_log_size":1238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22413312,"num_objects":342,"num_object_clones":0,"num_object_copies":1026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1238,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"27'1246","reported_seq":1282,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:09.681705+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:45:09.681705+0000","last_peered":"2024-09-11T14:45:09.681705+0000","last_clean":"2024-09-11T14:45:09.681705+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:45:09.681705+0000","last_undegraded":"2024-09-11T14:45:09.681705+0000","last_fullsized":"2024-09-11T14:45:09.681705+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1246,"log_dups_size":0,"ondisk_log_size":1246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23461888,"num_objects":358,"num_object_clones":0,"num_object_copies":1074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1246,"num_write_kb":51328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"27'1374","reported_seq":1410,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:09.467088+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:45:09.467088+0000","last_peered":"2024-09-11T14:45:09.467088+0000","last_clean":"2024-09-11T14:45:09.467088+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:45:09.467088+0000","last_undegraded":"2024-09-11T14:45:09.467088+0000","last_fullsized":"2024-09-11T14:45:09.467088+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1374,"log_dups_size":0,"ondisk_log_size":1374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25821184,"num_objects":394,"num_object_clones":0,"num_object_copies":1182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1374,"num_write_kb":56576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"27'1302","reported_seq":1338,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:09.537681+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:45:09.537681+0000","last_peered":"2024-09-11T14:45:09.537681+0000","last_clean":"2024-09-11T14:45:09.537681+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:45:09.537681+0000","last_undegraded":"2024-09-11T14:45:09.537681+0000","last_fullsized":"2024-09-11T14:45:09.537681+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1302,"log_dups_size":0,"ondisk_log_size":1302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24379392,"num_objects":372,"num_object_clones":0,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1302,"num_write_kb":53568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"27'1218","reported_seq":1254,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:09.279570+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:45:09.279570+0000","last_peered":"2024-09-11T14:45:09.279570+0000","last_clean":"2024-09-11T14:45:09.279570+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:45:09.279570+0000","last_undegraded":"2024-09-11T14:45:09.279570+0000","last_fullsized":"2024-09-11T14:45:09.279570+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1218,"log_dups_size":0,"ondisk_log_size":1218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24641536,"num_objects":376,"num_object_clones":0,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1218,"num_write_kb":51008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":121,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084543+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:42:13.084543+0000","last_peered":"2024-09-11T14:42:13.084543+0000","last_clean":"2024-09-11T14:42:13.084543+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:42:13.084543+0000","last_undegraded":"2024-09-11T14:42:13.084543+0000","last_fullsized":"2024-09-11T14:42:13.084543+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":50,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.184283+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:42:13.184283+0000","last_peered":"2024-09-11T14:42:13.184283+0000","last_clean":"2024-09-11T14:42:13.184283+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:42:13.184283+0000","last_undegraded":"2024-09-11T14:42:13.184283+0000","last_fullsized":"2024-09-11T14:42:13.184283+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"27'1250","reported_seq":1286,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:09.720938+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:45:09.720938+0000","last_peered":"2024-09-11T14:45:09.720938+0000","last_clean":"2024-09-11T14:45:09.720938+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:45:09.720938+0000","last_undegraded":"2024-09-11T14:45:09.720938+0000","last_fullsized":"2024-09-11T14:45:09.720938+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1250,"log_dups_size":0,"ondisk_log_size":1250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24379392,"num_objects":372,"num_object_clones":0,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1250,"num_write_kb":51904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.084333+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:42:13.084333+0000","last_peered":"2024-09-11T14:42:13.084333+0000","last_clean":"2024-09-11T14:42:13.084333+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:42:13.084333+0000","last_undegraded":"2024-09-11T14:42:13.084333+0000","last_fullsized":"2024-09-11T14:42:13.084333+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"27'1239","reported_seq":1275,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:09.335678+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:45:09.335678+0000","last_peered":"2024-09-11T14:45:09.335678+0000","last_clean":"2024-09-11T14:45:09.335678+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:45:09.335678+0000","last_undegraded":"2024-09-11T14:45:09.335678+0000","last_fullsized":"2024-09-11T14:45:09.335678+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1239,"log_dups_size":0,"ondisk_log_size":1239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624576,"num_objects":391,"num_object_clones":0,"num_object_copies":1173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1239,"num_write_kb":52160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.179633+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:42:13.179633+0000","last_peered":"2024-09-11T14:42:13.179633+0000","last_clean":"2024-09-11T14:42:13.179633+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:42:13.179633+0000","last_undegraded":"2024-09-11T14:42:13.179633+0000","last_fullsized":"2024-09-11T14:42:13.179633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"27'1249","reported_seq":1285,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:09.635354+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:45:09.635354+0000","last_peered":"2024-09-11T14:45:09.635354+0000","last_clean":"2024-09-11T14:45:09.635354+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:45:09.635354+0000","last_undegraded":"2024-09-11T14:45:09.635354+0000","last_fullsized":"2024-09-11T14:45:09.635354+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1249,"log_dups_size":0,"ondisk_log_size":1249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24707072,"num_objects":377,"num_object_clones":0,"num_object_copies":1131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1249,"num_write_kb":52032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":51,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.082423+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:42:13.082423+0000","last_peered":"2024-09-11T14:42:13.082423+0000","last_clean":"2024-09-11T14:42:13.082423+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:42:13.082423+0000","last_undegraded":"2024-09-11T14:42:13.082423+0000","last_fullsized":"2024-09-11T14:42:13.082423+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"27'1210","reported_seq":1246,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:05.849828+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:45:05.849828+0000","last_peered":"2024-09-11T14:45:05.849828+0000","last_clean":"2024-09-11T14:45:05.849828+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:45:05.849828+0000","last_undegraded":"2024-09-11T14:45:05.849828+0000","last_fullsized":"2024-09-11T14:45:05.849828+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1210,"log_dups_size":0,"ondisk_log_size":1210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25427968,"num_objects":388,"num_object_clones":0,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1210,"num_write_kb":51136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"27'1282","reported_seq":1318,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:09.649028+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:45:09.649028+0000","last_peered":"2024-09-11T14:45:09.649028+0000","last_clean":"2024-09-11T14:45:09.649028+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:45:09.649028+0000","last_undegraded":"2024-09-11T14:45:09.649028+0000","last_fullsized":"2024-09-11T14:45:09.649028+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1282,"log_dups_size":0,"ondisk_log_size":1282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24772608,"num_objects":378,"num_object_clones":0,"num_object_copies":1134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1282,"num_write_kb":53120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111507+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:42:13.111507+0000","last_peered":"2024-09-11T14:42:13.111507+0000","last_clean":"2024-09-11T14:42:13.111507+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:42:13.111507+0000","last_undegraded":"2024-09-11T14:42:13.111507+0000","last_fullsized":"2024-09-11T14:42:13.111507+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"27'1167","reported_seq":1203,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:05.305676+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:45:05.305676+0000","last_peered":"2024-09-11T14:45:05.305676+0000","last_clean":"2024-09-11T14:45:05.305676+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:45:05.305676+0000","last_undegraded":"2024-09-11T14:45:05.305676+0000","last_fullsized":"2024-09-11T14:45:05.305676+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1167,"log_dups_size":0,"ondisk_log_size":1167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22085632,"num_objects":337,"num_object_clones":0,"num_object_copies":1011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1167,"num_write_kb":48128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.086961+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:42:13.086961+0000","last_peered":"2024-09-11T14:42:13.086961+0000","last_clean":"2024-09-11T14:42:13.086961+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:42:13.086961+0000","last_undegraded":"2024-09-11T14:42:13.086961+0000","last_fullsized":"2024-09-11T14:42:13.086961+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"27'1284","reported_seq":1320,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:09.302646+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:45:09.302646+0000","last_peered":"2024-09-11T14:45:09.302646+0000","last_clean":"2024-09-11T14:45:09.302646+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:45:09.302646+0000","last_undegraded":"2024-09-11T14:45:09.302646+0000","last_fullsized":"2024-09-11T14:45:09.302646+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1284,"log_dups_size":0,"ondisk_log_size":1284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24248320,"num_objects":370,"num_object_clones":0,"num_object_copies":1110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1284,"num_write_kb":52928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.111171+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:42:13.111171+0000","last_peered":"2024-09-11T14:42:13.111171+0000","last_clean":"2024-09-11T14:42:13.111171+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:42:13.111171+0000","last_undegraded":"2024-09-11T14:42:13.111171+0000","last_fullsized":"2024-09-11T14:42:13.111171+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"27'1252","reported_seq":1288,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:45:05.697534+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:45:05.697534+0000","last_peered":"2024-09-11T14:45:05.697534+0000","last_clean":"2024-09-11T14:45:05.697534+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:45:05.697534+0000","last_undegraded":"2024-09-11T14:45:05.697534+0000","last_fullsized":"2024-09-11T14:45:05.697534+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1252,"log_dups_size":0,"ondisk_log_size":1252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24641536,"num_objects":376,"num_object_clones":0,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1252,"num_write_kb":52096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T14:42:13.181385+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:42:13.181385+0000","last_peered":"2024-09-11T14:42:13.181385+0000","last_clean":"2024-09-11T14:42:13.181385+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:42:13.181385+0000","last_undegraded":"2024-09-11T14:42:13.181385+0000","last_fullsized":"2024-09-11T14:42:13.181385+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":394985496,"num_objects":6028,"num_object_clones":0,"num_object_copies":18084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20180,"num_write_kb":838593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":211783245824,"internally_reserved":0,"allocated":2428248064,"data_stored":2428248064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20180,"ondisk_log_size":20180,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":211783245824,"internally_reserved":0,"allocated":2428248064,"data_stored":2428248064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":52857491456,"internally_reserved":0,"allocated":695382016,"data_stored":695382016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673199,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":565640,"kb_used_data":565640,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51732088,"statfs":{"total":53552873472,"available":52973658112,"internally_reserved":0,"allocated":579215360,"data_stored":579215360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705908,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":686468,"kb_used_data":686468,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51611260,"statfs":{"total":53552873472,"available":52849930240,"internally_reserved":0,"allocated":702943232,"data_stored":702943232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705908,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":603412,"kb_used_data":603412,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51694316,"statfs":{"total":53552873472,"available":52934979584,"internally_reserved":0,"allocated":617893888,"data_stored":617893888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705908,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":538432,"kb_used_data":538432,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51759296,"statfs":{"total":53552873472,"available":53001519104,"internally_reserved":0,"allocated":551354368,"data_stored":551354368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":52857491456,"internally_reserved":0,"allocated":695382016,"data_stored":695382016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53005897728,"internally_reserved":0,"allocated":546975744,"data_stored":546975744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":52941295616,"internally_reserved":0,"allocated":611577856,"data_stored":611577856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":52978561024,"internally_reserved":0,"allocated":574312448,"data_stored":574312448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":52857491456,"internally_reserved":0,"allocated":695382016,"data_stored":695382016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":53005897728,"internally_reserved":0,"allocated":546975744,"data_stored":546975744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":52941295616,"internally_reserved":0,"allocated":611577856,"data_stored":611577856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":52978561024,"internally_reserved":0,"allocated":574312448,"data_stored":574312448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":52857491456,"internally_reserved":0,"allocated":695382016,"data_stored":695382016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:45:12.816 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:45:12.981 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:12.977+0000 7f89532d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:12.986 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:12.985+0000 7f89532d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:15.514 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:45:15.655 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:15.651+0000 7f2ced911640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:15.659 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:15.658+0000 7f2ced911640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:17.445 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:45:17.445 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-11T14:45:17.617 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:17.603+0000 7f2265d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:17.622 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:17.621+0000 7f2265d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:17.930 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:45:18.111 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:18.106+0000 7f2ef6591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:18.118 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:18.116+0000 7f2ef6591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:18.225 INFO:teuthology.orchestra.run.smithi181.stderr:reweighted osd.0 to 1 (10000) 2024-09-11T14:45:18.246 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-11T14:45:18.415 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:18.401+0000 7fc497351640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:18.421 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:18.420+0000 7fc497351640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:19.231 INFO:teuthology.orchestra.run.smithi181.stderr:reweighted osd.1 to 1 (10000) 2024-09-11T14:45:19.250 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-11T14:45:19.410 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:19.398+0000 7f932e89c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:19.418 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:19.417+0000 7f932e89c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:20.237 INFO:teuthology.orchestra.run.smithi181.stderr:reweighted osd.2 to 1 (10000) 2024-09-11T14:45:20.258 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-11T14:45:20.426 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:20.415+0000 7efc58151640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:20.432 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:20.431+0000 7efc58151640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:20.562 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:45:20.708 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:20.704+0000 7f565a111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:20.713 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:20.712+0000 7f565a111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:21.248 INFO:teuthology.orchestra.run.smithi181.stderr:reweighted osd.3 to 1 (10000) 2024-09-11T14:45:21.273 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-11T14:45:21.274 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:45:21.453 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:21.444+0000 7f2d7f45a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:21.461 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:21.460+0000 7f2d7f45a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:21.838 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:45:21.838 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:45:21.858 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2024-09-11T14:45:21.247949+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":331285067,"num_objects":5051,"num_object_clones":0,"num_object_copies":15153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":21242,"num_write_kb":839701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21216,"ondisk_log_size":21216,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":2021936,"kb_used_data":2021936,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":207168976,"statfs":{"total":214211493888,"available":212141031424,"internally_reserved":0,"allocated":2070462464,"data_stored":2070462464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28770304,"num_objects":-439,"num_object_clones":0,"num_object_copies":-1317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":439,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.486399"},"pg_stats":[{"pgid":"3.f","version":"29'1290","reported_seq":1332,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.345888+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:45:19.345888+0000","last_peered":"2024-09-11T14:45:19.345888+0000","last_clean":"2024-09-11T14:45:19.345888+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:45:19.345888+0000","last_undegraded":"2024-09-11T14:45:19.345888+0000","last_fullsized":"2024-09-11T14:45:19.345888+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1290,"log_dups_size":0,"ondisk_log_size":1290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21495808,"num_objects":328,"num_object_clones":0,"num_object_copies":984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1290,"num_write_kb":51776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"30'1444","reported_seq":1489,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-11T14:45:20.857172+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:45:20.857172+0000","last_peered":"2024-09-11T14:45:20.857172+0000","last_clean":"2024-09-11T14:45:20.857172+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:45:20.857172+0000","last_undegraded":"2024-09-11T14:45:20.857172+0000","last_fullsized":"2024-09-11T14:45:20.857172+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1444,"log_dups_size":0,"ondisk_log_size":1444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22020096,"num_objects":336,"num_object_clones":0,"num_object_copies":1008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1444,"num_write_kb":56960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"28'1339","reported_seq":1382,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.470266+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:45:19.470266+0000","last_peered":"2024-09-11T14:45:19.470266+0000","last_clean":"2024-09-11T14:45:19.470266+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:45:19.470266+0000","last_undegraded":"2024-09-11T14:45:19.470266+0000","last_fullsized":"2024-09-11T14:45:19.470266+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1339,"log_dups_size":0,"ondisk_log_size":1339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21495832,"num_objects":329,"num_object_clones":0,"num_object_copies":987,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1339,"num_write_kb":53313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"29'1293","reported_seq":1335,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.341127+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:45:19.341127+0000","last_peered":"2024-09-11T14:45:19.341127+0000","last_clean":"2024-09-11T14:45:19.341127+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:45:19.341127+0000","last_undegraded":"2024-09-11T14:45:19.341127+0000","last_fullsized":"2024-09-11T14:45:19.341127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1293,"log_dups_size":0,"ondisk_log_size":1293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18808832,"num_objects":287,"num_object_clones":0,"num_object_copies":861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1293,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"29'1298","reported_seq":1340,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.296660+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:45:19.296660+0000","last_peered":"2024-09-11T14:45:19.296660+0000","last_clean":"2024-09-11T14:45:19.296660+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:45:19.296660+0000","last_undegraded":"2024-09-11T14:45:19.296660+0000","last_fullsized":"2024-09-11T14:45:19.296660+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1298,"log_dups_size":0,"ondisk_log_size":1298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20054016,"num_objects":306,"num_object_clones":0,"num_object_copies":918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1298,"num_write_kb":51328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"29'1431","reported_seq":1473,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.739374+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:45:19.739374+0000","last_peered":"2024-09-11T14:45:19.739374+0000","last_clean":"2024-09-11T14:45:19.739374+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:45:19.739374+0000","last_undegraded":"2024-09-11T14:45:19.739374+0000","last_fullsized":"2024-09-11T14:45:19.739374+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1431,"log_dups_size":0,"ondisk_log_size":1431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22085632,"num_objects":337,"num_object_clones":0,"num_object_copies":1011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1431,"num_write_kb":56576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"29'1349","reported_seq":1391,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.700633+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:45:19.700633+0000","last_peered":"2024-09-11T14:45:19.700633+0000","last_clean":"2024-09-11T14:45:19.700633+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:45:19.700633+0000","last_undegraded":"2024-09-11T14:45:19.700633+0000","last_fullsized":"2024-09-11T14:45:19.700633+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1349,"log_dups_size":0,"ondisk_log_size":1349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21299200,"num_objects":325,"num_object_clones":0,"num_object_copies":975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1349,"num_write_kb":53568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"29'1284","reported_seq":1326,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.811486+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:45:19.811486+0000","last_peered":"2024-09-11T14:45:19.811486+0000","last_clean":"2024-09-11T14:45:19.811486+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:45:19.811486+0000","last_undegraded":"2024-09-11T14:45:19.811486+0000","last_fullsized":"2024-09-11T14:45:19.811486+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1284,"log_dups_size":0,"ondisk_log_size":1284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20316160,"num_objects":310,"num_object_clones":0,"num_object_copies":930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1284,"num_write_kb":51008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":127,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.283105+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:45:19.283105+0000","last_peered":"2024-09-11T14:45:19.283105+0000","last_clean":"2024-09-11T14:45:19.283105+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:45:19.283105+0000","last_undegraded":"2024-09-11T14:45:19.283105+0000","last_fullsized":"2024-09-11T14:45:19.283105+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":56,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.481377+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:45:19.481377+0000","last_peered":"2024-09-11T14:45:19.481377+0000","last_clean":"2024-09-11T14:45:19.481377+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:45:19.481377+0000","last_undegraded":"2024-09-11T14:45:19.481377+0000","last_fullsized":"2024-09-11T14:45:19.481377+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"29'1307","reported_seq":1349,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.705799+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:45:19.705799+0000","last_peered":"2024-09-11T14:45:19.705799+0000","last_clean":"2024-09-11T14:45:19.705799+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:45:19.705799+0000","last_undegraded":"2024-09-11T14:45:19.705799+0000","last_fullsized":"2024-09-11T14:45:19.705799+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1307,"log_dups_size":0,"ondisk_log_size":1307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20643840,"num_objects":315,"num_object_clones":0,"num_object_copies":945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1307,"num_write_kb":51904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":55,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.343719+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:45:19.343719+0000","last_peered":"2024-09-11T14:45:19.343719+0000","last_clean":"2024-09-11T14:45:19.343719+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:45:19.343719+0000","last_undegraded":"2024-09-11T14:45:19.343719+0000","last_fullsized":"2024-09-11T14:45:19.343719+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"29'1299","reported_seq":1341,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.699571+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:45:19.699571+0000","last_peered":"2024-09-11T14:45:19.699571+0000","last_clean":"2024-09-11T14:45:19.699571+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:45:19.699571+0000","last_undegraded":"2024-09-11T14:45:19.699571+0000","last_fullsized":"2024-09-11T14:45:19.699571+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1299,"log_dups_size":0,"ondisk_log_size":1299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21692416,"num_objects":331,"num_object_clones":0,"num_object_copies":993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1299,"num_write_kb":52160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":55,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.462887+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:45:19.462887+0000","last_peered":"2024-09-11T14:45:19.462887+0000","last_clean":"2024-09-11T14:45:19.462887+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:45:19.462887+0000","last_undegraded":"2024-09-11T14:45:19.462887+0000","last_fullsized":"2024-09-11T14:45:19.462887+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"29'1308","reported_seq":1350,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.840357+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:45:19.840357+0000","last_peered":"2024-09-11T14:45:19.840357+0000","last_clean":"2024-09-11T14:45:19.840357+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:45:19.840357+0000","last_undegraded":"2024-09-11T14:45:19.840357+0000","last_fullsized":"2024-09-11T14:45:19.840357+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1308,"log_dups_size":0,"ondisk_log_size":1308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20840448,"num_objects":318,"num_object_clones":0,"num_object_copies":954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1308,"num_write_kb":52032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":57,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.356061+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:45:19.356061+0000","last_peered":"2024-09-11T14:45:19.356061+0000","last_clean":"2024-09-11T14:45:19.356061+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:45:19.356061+0000","last_undegraded":"2024-09-11T14:45:19.356061+0000","last_fullsized":"2024-09-11T14:45:19.356061+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"30'1286","reported_seq":1331,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-11T14:45:20.447303+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:45:20.447303+0000","last_peered":"2024-09-11T14:45:20.447303+0000","last_clean":"2024-09-11T14:45:20.447303+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:45:20.447303+0000","last_undegraded":"2024-09-11T14:45:20.447303+0000","last_fullsized":"2024-09-11T14:45:20.447303+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1286,"log_dups_size":0,"ondisk_log_size":1286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20447232,"num_objects":312,"num_object_clones":0,"num_object_copies":936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1286,"num_write_kb":51136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"29'1328","reported_seq":1370,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.757191+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:45:19.757191+0000","last_peered":"2024-09-11T14:45:19.757191+0000","last_clean":"2024-09-11T14:45:19.757191+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:45:19.757191+0000","last_undegraded":"2024-09-11T14:45:19.757191+0000","last_fullsized":"2024-09-11T14:45:19.757191+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1328,"log_dups_size":0,"ondisk_log_size":1328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21757952,"num_objects":332,"num_object_clones":0,"num_object_copies":996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1328,"num_write_kb":53120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":58,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-11T14:45:20.312944+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:45:20.312944+0000","last_peered":"2024-09-11T14:45:20.312944+0000","last_clean":"2024-09-11T14:45:20.312944+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:45:20.312944+0000","last_undegraded":"2024-09-11T14:45:20.312944+0000","last_fullsized":"2024-09-11T14:45:20.312944+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"30'1231","reported_seq":1276,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-11T14:45:20.597395+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:45:20.597395+0000","last_peered":"2024-09-11T14:45:20.597395+0000","last_clean":"2024-09-11T14:45:20.597395+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:45:20.597395+0000","last_undegraded":"2024-09-11T14:45:20.597395+0000","last_fullsized":"2024-09-11T14:45:20.597395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1231,"log_dups_size":0,"ondisk_log_size":1231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17891328,"num_objects":273,"num_object_clones":0,"num_object_copies":819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1231,"num_write_kb":48128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":55,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.282603+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:45:19.282603+0000","last_peered":"2024-09-11T14:45:19.282603+0000","last_clean":"2024-09-11T14:45:19.282603+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:45:19.282603+0000","last_undegraded":"2024-09-11T14:45:19.282603+0000","last_fullsized":"2024-09-11T14:45:19.282603+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"29'1333","reported_seq":1375,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.640187+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:45:19.640187+0000","last_peered":"2024-09-11T14:45:19.640187+0000","last_clean":"2024-09-11T14:45:19.640187+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:45:19.640187+0000","last_undegraded":"2024-09-11T14:45:19.640187+0000","last_fullsized":"2024-09-11T14:45:19.640187+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1333,"log_dups_size":0,"ondisk_log_size":1333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21037056,"num_objects":321,"num_object_clones":0,"num_object_copies":963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1333,"num_write_kb":52928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":58,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-11T14:45:20.302271+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:45:20.302271+0000","last_peered":"2024-09-11T14:45:20.302271+0000","last_clean":"2024-09-11T14:45:20.302271+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:45:20.302271+0000","last_undegraded":"2024-09-11T14:45:20.302271+0000","last_fullsized":"2024-09-11T14:45:20.302271+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"30'1341","reported_seq":1386,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-11T14:45:20.620160+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:45:20.620160+0000","last_peered":"2024-09-11T14:45:20.620160+0000","last_clean":"2024-09-11T14:45:20.620160+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:45:20.620160+0000","last_undegraded":"2024-09-11T14:45:20.620160+0000","last_fullsized":"2024-09-11T14:45:20.620160+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1341,"log_dups_size":0,"ondisk_log_size":1341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18808832,"num_objects":287,"num_object_clones":0,"num_object_copies":861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1341,"num_write_kb":52096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":55,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.478301+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:45:19.478301+0000","last_peered":"2024-09-11T14:45:19.478301+0000","last_clean":"2024-09-11T14:45:19.478301+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:45:19.478301+0000","last_undegraded":"2024-09-11T14:45:19.478301+0000","last_fullsized":"2024-09-11T14:45:19.478301+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":330694680,"num_objects":5047,"num_object_clones":0,"num_object_copies":15141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":21161,"num_write_kb":838593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":212163764224,"internally_reserved":0,"allocated":2047729664,"data_stored":2047729664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21161,"ondisk_log_size":21161,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":212163633152,"internally_reserved":0,"allocated":2047860736,"data_stored":2047860736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":52957089792,"internally_reserved":0,"allocated":595783680,"data_stored":595783680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673214,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":453564,"kb_used_data":453564,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51844164,"statfs":{"total":53552873472,"available":53088423936,"internally_reserved":0,"allocated":464449536,"data_stored":464449536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":588692,"kb_used_data":588692,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51709036,"statfs":{"total":53552873472,"available":52950052864,"internally_reserved":0,"allocated":602820608,"data_stored":602820608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":517736,"kb_used_data":517736,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51779992,"statfs":{"total":53552873472,"available":53022711808,"internally_reserved":0,"allocated":530161664,"data_stored":530161664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":461944,"kb_used_data":461944,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51835784,"statfs":{"total":53552873472,"available":53079842816,"internally_reserved":0,"allocated":473030656,"data_stored":473030656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":52957089792,"internally_reserved":0,"allocated":595783680,"data_stored":595783680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53084065792,"internally_reserved":0,"allocated":468807680,"data_stored":468807680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53029306368,"internally_reserved":0,"allocated":523567104,"data_stored":523567104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53093171200,"internally_reserved":0,"allocated":459702272,"data_stored":459702272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":52957089792,"internally_reserved":0,"allocated":595783680,"data_stored":595783680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":53084065792,"internally_reserved":0,"allocated":468807680,"data_stored":468807680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":53029437440,"internally_reserved":0,"allocated":523436032,"data_stored":523436032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":53093171200,"internally_reserved":0,"allocated":459702272,"data_stored":459702272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":52957089792,"internally_reserved":0,"allocated":595783680,"data_stored":595783680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:45:21.860 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:45:22.005 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:21.997+0000 7f70c6b11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:22.010 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:22.009+0000 7f70c6b11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:22.388 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:45:22.388 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:45:22.413 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2024-09-11T14:45:21.247949+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":331285067,"num_objects":5051,"num_object_clones":0,"num_object_copies":15153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":21242,"num_write_kb":839701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21216,"ondisk_log_size":21216,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":2021936,"kb_used_data":2021936,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":207168976,"statfs":{"total":214211493888,"available":212141031424,"internally_reserved":0,"allocated":2070462464,"data_stored":2070462464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28770304,"num_objects":-439,"num_object_clones":0,"num_object_copies":-1317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":439,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.486399"},"pg_stats":[{"pgid":"3.f","version":"29'1290","reported_seq":1332,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.345888+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:45:19.345888+0000","last_peered":"2024-09-11T14:45:19.345888+0000","last_clean":"2024-09-11T14:45:19.345888+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:45:19.345888+0000","last_undegraded":"2024-09-11T14:45:19.345888+0000","last_fullsized":"2024-09-11T14:45:19.345888+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1290,"log_dups_size":0,"ondisk_log_size":1290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21495808,"num_objects":328,"num_object_clones":0,"num_object_copies":984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1290,"num_write_kb":51776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"30'1444","reported_seq":1489,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-11T14:45:20.857172+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:45:20.857172+0000","last_peered":"2024-09-11T14:45:20.857172+0000","last_clean":"2024-09-11T14:45:20.857172+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:45:20.857172+0000","last_undegraded":"2024-09-11T14:45:20.857172+0000","last_fullsized":"2024-09-11T14:45:20.857172+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1444,"log_dups_size":0,"ondisk_log_size":1444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22020096,"num_objects":336,"num_object_clones":0,"num_object_copies":1008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1444,"num_write_kb":56960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"28'1339","reported_seq":1382,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.470266+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:45:19.470266+0000","last_peered":"2024-09-11T14:45:19.470266+0000","last_clean":"2024-09-11T14:45:19.470266+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:45:19.470266+0000","last_undegraded":"2024-09-11T14:45:19.470266+0000","last_fullsized":"2024-09-11T14:45:19.470266+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1339,"log_dups_size":0,"ondisk_log_size":1339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21495832,"num_objects":329,"num_object_clones":0,"num_object_copies":987,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1339,"num_write_kb":53313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"29'1293","reported_seq":1335,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.341127+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:45:19.341127+0000","last_peered":"2024-09-11T14:45:19.341127+0000","last_clean":"2024-09-11T14:45:19.341127+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:45:19.341127+0000","last_undegraded":"2024-09-11T14:45:19.341127+0000","last_fullsized":"2024-09-11T14:45:19.341127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1293,"log_dups_size":0,"ondisk_log_size":1293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18808832,"num_objects":287,"num_object_clones":0,"num_object_copies":861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1293,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"29'1298","reported_seq":1340,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.296660+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:45:19.296660+0000","last_peered":"2024-09-11T14:45:19.296660+0000","last_clean":"2024-09-11T14:45:19.296660+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:45:19.296660+0000","last_undegraded":"2024-09-11T14:45:19.296660+0000","last_fullsized":"2024-09-11T14:45:19.296660+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1298,"log_dups_size":0,"ondisk_log_size":1298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20054016,"num_objects":306,"num_object_clones":0,"num_object_copies":918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1298,"num_write_kb":51328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"29'1431","reported_seq":1473,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.739374+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:45:19.739374+0000","last_peered":"2024-09-11T14:45:19.739374+0000","last_clean":"2024-09-11T14:45:19.739374+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:45:19.739374+0000","last_undegraded":"2024-09-11T14:45:19.739374+0000","last_fullsized":"2024-09-11T14:45:19.739374+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1431,"log_dups_size":0,"ondisk_log_size":1431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22085632,"num_objects":337,"num_object_clones":0,"num_object_copies":1011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1431,"num_write_kb":56576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"29'1349","reported_seq":1391,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.700633+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:45:19.700633+0000","last_peered":"2024-09-11T14:45:19.700633+0000","last_clean":"2024-09-11T14:45:19.700633+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:45:19.700633+0000","last_undegraded":"2024-09-11T14:45:19.700633+0000","last_fullsized":"2024-09-11T14:45:19.700633+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1349,"log_dups_size":0,"ondisk_log_size":1349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21299200,"num_objects":325,"num_object_clones":0,"num_object_copies":975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1349,"num_write_kb":53568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"29'1284","reported_seq":1326,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.811486+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:45:19.811486+0000","last_peered":"2024-09-11T14:45:19.811486+0000","last_clean":"2024-09-11T14:45:19.811486+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:45:19.811486+0000","last_undegraded":"2024-09-11T14:45:19.811486+0000","last_fullsized":"2024-09-11T14:45:19.811486+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1284,"log_dups_size":0,"ondisk_log_size":1284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20316160,"num_objects":310,"num_object_clones":0,"num_object_copies":930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1284,"num_write_kb":51008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":127,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.283105+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:45:19.283105+0000","last_peered":"2024-09-11T14:45:19.283105+0000","last_clean":"2024-09-11T14:45:19.283105+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:45:19.283105+0000","last_undegraded":"2024-09-11T14:45:19.283105+0000","last_fullsized":"2024-09-11T14:45:19.283105+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":56,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.481377+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:45:19.481377+0000","last_peered":"2024-09-11T14:45:19.481377+0000","last_clean":"2024-09-11T14:45:19.481377+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:45:19.481377+0000","last_undegraded":"2024-09-11T14:45:19.481377+0000","last_fullsized":"2024-09-11T14:45:19.481377+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"29'1307","reported_seq":1349,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.705799+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:45:19.705799+0000","last_peered":"2024-09-11T14:45:19.705799+0000","last_clean":"2024-09-11T14:45:19.705799+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:45:19.705799+0000","last_undegraded":"2024-09-11T14:45:19.705799+0000","last_fullsized":"2024-09-11T14:45:19.705799+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1307,"log_dups_size":0,"ondisk_log_size":1307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20643840,"num_objects":315,"num_object_clones":0,"num_object_copies":945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1307,"num_write_kb":51904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":55,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.343719+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:45:19.343719+0000","last_peered":"2024-09-11T14:45:19.343719+0000","last_clean":"2024-09-11T14:45:19.343719+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:45:19.343719+0000","last_undegraded":"2024-09-11T14:45:19.343719+0000","last_fullsized":"2024-09-11T14:45:19.343719+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"29'1299","reported_seq":1341,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.699571+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:45:19.699571+0000","last_peered":"2024-09-11T14:45:19.699571+0000","last_clean":"2024-09-11T14:45:19.699571+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:45:19.699571+0000","last_undegraded":"2024-09-11T14:45:19.699571+0000","last_fullsized":"2024-09-11T14:45:19.699571+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1299,"log_dups_size":0,"ondisk_log_size":1299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21692416,"num_objects":331,"num_object_clones":0,"num_object_copies":993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1299,"num_write_kb":52160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":55,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.462887+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:45:19.462887+0000","last_peered":"2024-09-11T14:45:19.462887+0000","last_clean":"2024-09-11T14:45:19.462887+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:45:19.462887+0000","last_undegraded":"2024-09-11T14:45:19.462887+0000","last_fullsized":"2024-09-11T14:45:19.462887+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"29'1308","reported_seq":1350,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.840357+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:45:19.840357+0000","last_peered":"2024-09-11T14:45:19.840357+0000","last_clean":"2024-09-11T14:45:19.840357+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:45:19.840357+0000","last_undegraded":"2024-09-11T14:45:19.840357+0000","last_fullsized":"2024-09-11T14:45:19.840357+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1308,"log_dups_size":0,"ondisk_log_size":1308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20840448,"num_objects":318,"num_object_clones":0,"num_object_copies":954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1308,"num_write_kb":52032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":57,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.356061+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:45:19.356061+0000","last_peered":"2024-09-11T14:45:19.356061+0000","last_clean":"2024-09-11T14:45:19.356061+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:45:19.356061+0000","last_undegraded":"2024-09-11T14:45:19.356061+0000","last_fullsized":"2024-09-11T14:45:19.356061+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"30'1286","reported_seq":1331,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-11T14:45:20.447303+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:45:20.447303+0000","last_peered":"2024-09-11T14:45:20.447303+0000","last_clean":"2024-09-11T14:45:20.447303+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:45:20.447303+0000","last_undegraded":"2024-09-11T14:45:20.447303+0000","last_fullsized":"2024-09-11T14:45:20.447303+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1286,"log_dups_size":0,"ondisk_log_size":1286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20447232,"num_objects":312,"num_object_clones":0,"num_object_copies":936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1286,"num_write_kb":51136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"29'1328","reported_seq":1370,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.757191+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:45:19.757191+0000","last_peered":"2024-09-11T14:45:19.757191+0000","last_clean":"2024-09-11T14:45:19.757191+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:45:19.757191+0000","last_undegraded":"2024-09-11T14:45:19.757191+0000","last_fullsized":"2024-09-11T14:45:19.757191+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1328,"log_dups_size":0,"ondisk_log_size":1328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21757952,"num_objects":332,"num_object_clones":0,"num_object_copies":996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1328,"num_write_kb":53120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":58,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-11T14:45:20.312944+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:45:20.312944+0000","last_peered":"2024-09-11T14:45:20.312944+0000","last_clean":"2024-09-11T14:45:20.312944+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:45:20.312944+0000","last_undegraded":"2024-09-11T14:45:20.312944+0000","last_fullsized":"2024-09-11T14:45:20.312944+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"30'1231","reported_seq":1276,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-11T14:45:20.597395+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:45:20.597395+0000","last_peered":"2024-09-11T14:45:20.597395+0000","last_clean":"2024-09-11T14:45:20.597395+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:45:20.597395+0000","last_undegraded":"2024-09-11T14:45:20.597395+0000","last_fullsized":"2024-09-11T14:45:20.597395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1231,"log_dups_size":0,"ondisk_log_size":1231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17891328,"num_objects":273,"num_object_clones":0,"num_object_copies":819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1231,"num_write_kb":48128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":55,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.282603+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:45:19.282603+0000","last_peered":"2024-09-11T14:45:19.282603+0000","last_clean":"2024-09-11T14:45:19.282603+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:45:19.282603+0000","last_undegraded":"2024-09-11T14:45:19.282603+0000","last_fullsized":"2024-09-11T14:45:19.282603+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"29'1333","reported_seq":1375,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.640187+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:45:19.640187+0000","last_peered":"2024-09-11T14:45:19.640187+0000","last_clean":"2024-09-11T14:45:19.640187+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:45:19.640187+0000","last_undegraded":"2024-09-11T14:45:19.640187+0000","last_fullsized":"2024-09-11T14:45:19.640187+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1333,"log_dups_size":0,"ondisk_log_size":1333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21037056,"num_objects":321,"num_object_clones":0,"num_object_copies":963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1333,"num_write_kb":52928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":58,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-11T14:45:20.302271+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:45:20.302271+0000","last_peered":"2024-09-11T14:45:20.302271+0000","last_clean":"2024-09-11T14:45:20.302271+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:45:20.302271+0000","last_undegraded":"2024-09-11T14:45:20.302271+0000","last_fullsized":"2024-09-11T14:45:20.302271+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"30'1341","reported_seq":1386,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-11T14:45:20.620160+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:45:20.620160+0000","last_peered":"2024-09-11T14:45:20.620160+0000","last_clean":"2024-09-11T14:45:20.620160+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:45:20.620160+0000","last_undegraded":"2024-09-11T14:45:20.620160+0000","last_fullsized":"2024-09-11T14:45:20.620160+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1341,"log_dups_size":0,"ondisk_log_size":1341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18808832,"num_objects":287,"num_object_clones":0,"num_object_copies":861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1341,"num_write_kb":52096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":55,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.478301+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:45:19.478301+0000","last_peered":"2024-09-11T14:45:19.478301+0000","last_clean":"2024-09-11T14:45:19.478301+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:45:19.478301+0000","last_undegraded":"2024-09-11T14:45:19.478301+0000","last_fullsized":"2024-09-11T14:45:19.478301+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":330694680,"num_objects":5047,"num_object_clones":0,"num_object_copies":15141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":21161,"num_write_kb":838593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":212163764224,"internally_reserved":0,"allocated":2047729664,"data_stored":2047729664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21161,"ondisk_log_size":21161,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":212163633152,"internally_reserved":0,"allocated":2047860736,"data_stored":2047860736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":52957089792,"internally_reserved":0,"allocated":595783680,"data_stored":595783680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673214,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":453564,"kb_used_data":453564,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51844164,"statfs":{"total":53552873472,"available":53088423936,"internally_reserved":0,"allocated":464449536,"data_stored":464449536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":588692,"kb_used_data":588692,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51709036,"statfs":{"total":53552873472,"available":52950052864,"internally_reserved":0,"allocated":602820608,"data_stored":602820608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":517736,"kb_used_data":517736,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51779992,"statfs":{"total":53552873472,"available":53022711808,"internally_reserved":0,"allocated":530161664,"data_stored":530161664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":461944,"kb_used_data":461944,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51835784,"statfs":{"total":53552873472,"available":53079842816,"internally_reserved":0,"allocated":473030656,"data_stored":473030656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":52957089792,"internally_reserved":0,"allocated":595783680,"data_stored":595783680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53084065792,"internally_reserved":0,"allocated":468807680,"data_stored":468807680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53029306368,"internally_reserved":0,"allocated":523567104,"data_stored":523567104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53093171200,"internally_reserved":0,"allocated":459702272,"data_stored":459702272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":52957089792,"internally_reserved":0,"allocated":595783680,"data_stored":595783680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":53084065792,"internally_reserved":0,"allocated":468807680,"data_stored":468807680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":53029437440,"internally_reserved":0,"allocated":523436032,"data_stored":523436032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":53093171200,"internally_reserved":0,"allocated":459702272,"data_stored":459702272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":52957089792,"internally_reserved":0,"allocated":595783680,"data_stored":595783680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:45:22.415 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-11T14:45:22.415 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:45:22.415 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-11T14:45:22.415 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-11T14:45:22.603 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:22.591+0000 7fb2b6111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:22.610 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:22.609+0000 7fb2b6111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:22.686 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:45:22.686 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:45:22.686 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:45:22.686 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:45:22.686 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:45:22.686 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:45:22.686 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:45:22.686 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:45:22.686 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:45:22.687 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:45:22.687 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:45:22.687 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:45:22.691 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:45:22.691 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:45:22.691 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-11T14:45:22.889 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:22.877+0000 7f107ced1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:22.894 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:22.893+0000 7f107ced1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:23.027 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:45:23.027 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:45:23.027 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:45:23.027 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:45:23.027 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:45:23.027 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:45:23.027 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:45:23.027 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:45:23.027 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:45:23.028 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:45:23.028 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:45:23.028 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:45:23.030 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:45:23.030 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:45:23.030 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-11T14:45:23.078 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:45:23.182 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:23.173+0000 7f2027191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:23.188 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:23.187+0000 7f2027191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:23.243 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:23.240+0000 7fc939751640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:23.249 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:23.247+0000 7fc939751640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:23.304 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:45:23.305 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:45:23.305 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:45:23.305 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:45:23.305 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:45:23.305 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:45:23.305 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:45:23.305 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:45:23.305 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:45:23.305 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:45:23.305 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:45:23.305 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:45:23.309 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:45:23.309 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:45:23.309 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-11T14:45:23.445 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:23.430+0000 7f73104bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:23.452 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:23.451+0000 7f73104bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:23.478 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:45:23.478 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:45:23.478 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:45:23.478 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:45:23.478 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:45:23.478 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:45:23.478 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:45:23.478 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:45:23.478 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:45:23.478 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:45:23.478 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:45:23.479 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:45:23.482 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:45:23.482 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:45:25.653 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:45:25.794 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:25.790+0000 7fe7cdd91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:25.802 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:25.801+0000 7fe7cdd91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:28.342 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:45:28.480 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:28.477+0000 7f6fa3b51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:28.483 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:45:28.484 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:45:28.484 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-11T14:45:28.484 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-11T14:45:28.487 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:28.486+0000 7f6fa3b51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:28.621 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:28.612+0000 7fd196e4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:28.626 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:28.624+0000 7fd196e4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:28.835 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:45:28.836 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:45:28.836 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:45:28.836 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:45:28.836 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:45:28.836 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:45:28.836 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:45:28.836 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:45:28.836 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:45:28.836 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:45:28.836 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:45:28.836 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:45:28.839 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:45:28.839 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:45:28.839 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-11T14:45:29.001 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:28.992+0000 7fcc50b51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:29.007 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:29.006+0000 7fcc50b51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:29.165 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:45:29.166 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:45:29.166 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:45:29.166 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:45:29.166 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:45:29.166 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:45:29.166 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:45:29.166 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:45:29.166 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:45:29.166 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:45:29.166 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:45:29.166 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:45:29.170 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:45:29.171 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:45:29.171 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-11T14:45:29.320 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:29.312+0000 7f2bfbb91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:29.326 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:29.324+0000 7f2bfbb91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:29.344 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:45:29.345 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:45:29.345 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:45:29.345 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:45:29.345 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:45:29.345 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:45:29.345 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:45:29.345 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:45:29.345 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:45:29.345 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:45:29.345 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:45:29.345 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:45:29.348 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:45:29.348 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:45:29.349 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-11T14:45:29.490 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:29.480+0000 7fd989cc8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:29.497 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:29.496+0000 7fd989cc8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:29.609 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:45:29.610 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:45:29.610 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:45:29.610 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:45:29.610 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:45:29.610 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:45:29.610 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:45:29.610 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:45:29.610 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:45:29.610 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:45:29.610 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:45:29.610 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:45:29.613 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:45:29.614 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:45:31.037 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:45:31.201 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:31.197+0000 7f8154efd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:31.209 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:31.208+0000 7f8154efd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:33.601 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:45:33.741 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:33.738+0000 7fab0caec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:33.747 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:33.746+0000 7fab0caec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:34.614 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:45:34.615 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:45:34.615 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-11T14:45:34.615 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-11T14:45:34.772 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:34.762+0000 7fb9eb663640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:34.778 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:34.776+0000 7fb9eb663640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:34.794 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:45:34.794 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:45:34.794 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:45:34.794 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:45:34.794 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:45:34.794 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:45:34.794 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:45:34.794 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:45:34.794 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:45:34.794 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:45:34.794 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:45:34.794 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:45:34.798 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:45:34.798 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:45:34.798 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-11T14:45:34.935 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:34.927+0000 7f9759ed1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:34.939 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:34.938+0000 7f9759ed1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:35.014 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:45:35.014 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:45:35.014 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:45:35.014 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:45:35.014 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:45:35.014 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:45:35.014 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:45:35.014 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:45:35.014 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:45:35.014 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:45:35.014 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:45:35.014 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:45:35.019 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:45:35.019 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:45:35.020 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-11T14:45:35.175 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:35.166+0000 7f750207d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:35.180 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:35.179+0000 7f750207d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:35.207 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:45:35.207 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:45:35.207 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:45:35.207 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:45:35.207 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:45:35.207 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:45:35.207 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:45:35.208 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:45:35.208 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:45:35.208 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:45:35.208 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:45:35.208 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:45:35.210 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:45:35.210 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:45:35.211 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-11T14:45:35.363 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:35.354+0000 7f5c42ea3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:35.370 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:35.368+0000 7f5c42ea3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:35.516 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:45:35.516 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:45:35.516 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:45:35.516 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:45:35.516 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:45:35.517 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:45:35.517 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:45:35.517 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:45:35.517 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:45:35.517 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:45:35.517 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:45:35.517 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:45:35.520 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:45:35.520 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:45:36.227 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:45:36.349 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:36.345+0000 7fe0d9391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:36.353 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:36.352+0000 7fe0d9391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:38.724 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:45:38.851 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:38.847+0000 7f0f86591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:38.858 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:38.856+0000 7f0f86591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:40.520 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:45:40.521 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:45:40.521 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:45:40.663 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:40.656+0000 7f5c2805c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:40.668 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:40.666+0000 7f5c2805c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:40.895 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:45:40.895 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:45:40.911 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2024-09-11T14:45:40.767156+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":229573195,"num_objects":3499,"num_object_clones":0,"num_object_copies":10497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":22794,"num_write_kb":839701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22768,"ondisk_log_size":22768,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":1413676,"kb_used_data":1413676,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":207777236,"statfs":{"total":214211493888,"available":212763889664,"internally_reserved":0,"allocated":1447604224,"data_stored":1447604224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-74645504,"num_objects":-1139,"num_object_clones":0,"num_object_copies":-3417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1139,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002716"},"pg_stats":[{"pgid":"3.f","version":"31'1406","reported_seq":1454,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:39.307769+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:45:39.307769+0000","last_peered":"2024-09-11T14:45:39.307769+0000","last_clean":"2024-09-11T14:45:39.307769+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:45:39.307769+0000","last_undegraded":"2024-09-11T14:45:39.307769+0000","last_fullsized":"2024-09-11T14:45:39.307769+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1406,"log_dups_size":0,"ondisk_log_size":1406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13893632,"num_objects":212,"num_object_clones":0,"num_object_copies":636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1406,"num_write_kb":51776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"31'1531","reported_seq":1579,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:35.756414+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:45:35.756414+0000","last_peered":"2024-09-11T14:45:35.756414+0000","last_clean":"2024-09-11T14:45:35.756414+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:45:35.756414+0000","last_undegraded":"2024-09-11T14:45:35.756414+0000","last_fullsized":"2024-09-11T14:45:35.756414+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1531,"log_dups_size":0,"ondisk_log_size":1531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16318464,"num_objects":249,"num_object_clones":0,"num_object_copies":747,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1531,"num_write_kb":56960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"31'1449","reported_seq":1498,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:39.592131+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:45:39.592131+0000","last_peered":"2024-09-11T14:45:39.592131+0000","last_clean":"2024-09-11T14:45:39.592131+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:45:39.592131+0000","last_undegraded":"2024-09-11T14:45:39.592131+0000","last_fullsized":"2024-09-11T14:45:39.592131+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1449,"log_dups_size":0,"ondisk_log_size":1449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14286872,"num_objects":219,"num_object_clones":0,"num_object_copies":657,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1449,"num_write_kb":53313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"31'1396","reported_seq":1444,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:39.442891+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:45:39.442891+0000","last_peered":"2024-09-11T14:45:39.442891+0000","last_clean":"2024-09-11T14:45:39.442891+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:45:39.442891+0000","last_undegraded":"2024-09-11T14:45:39.442891+0000","last_fullsized":"2024-09-11T14:45:39.442891+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1396,"log_dups_size":0,"ondisk_log_size":1396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12058624,"num_objects":184,"num_object_clones":0,"num_object_copies":552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1396,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"31'1385","reported_seq":1433,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:39.401493+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:45:39.401493+0000","last_peered":"2024-09-11T14:45:39.401493+0000","last_clean":"2024-09-11T14:45:39.401493+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:45:39.401493+0000","last_undegraded":"2024-09-11T14:45:39.401493+0000","last_fullsized":"2024-09-11T14:45:39.401493+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1385,"log_dups_size":0,"ondisk_log_size":1385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14352384,"num_objects":219,"num_object_clones":0,"num_object_copies":657,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1385,"num_write_kb":51328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"31'1539","reported_seq":1587,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:39.696450+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:45:39.696450+0000","last_peered":"2024-09-11T14:45:39.696450+0000","last_clean":"2024-09-11T14:45:39.696450+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:45:39.696450+0000","last_undegraded":"2024-09-11T14:45:39.696450+0000","last_fullsized":"2024-09-11T14:45:39.696450+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1539,"log_dups_size":0,"ondisk_log_size":1539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15007744,"num_objects":229,"num_object_clones":0,"num_object_copies":687,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1539,"num_write_kb":56576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"31'1459","reported_seq":1507,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:39.771620+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:45:39.771620+0000","last_peered":"2024-09-11T14:45:39.771620+0000","last_clean":"2024-09-11T14:45:39.771620+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:45:39.771620+0000","last_undegraded":"2024-09-11T14:45:39.771620+0000","last_fullsized":"2024-09-11T14:45:39.771620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1459,"log_dups_size":0,"ondisk_log_size":1459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14090240,"num_objects":215,"num_object_clones":0,"num_object_copies":645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1459,"num_write_kb":53568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"31'1390","reported_seq":1438,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:39.623180+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:45:39.623180+0000","last_peered":"2024-09-11T14:45:39.623180+0000","last_clean":"2024-09-11T14:45:39.623180+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:45:39.623180+0000","last_undegraded":"2024-09-11T14:45:39.623180+0000","last_fullsized":"2024-09-11T14:45:39.623180+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1390,"log_dups_size":0,"ondisk_log_size":1390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13369344,"num_objects":204,"num_object_clones":0,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1390,"num_write_kb":51008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":133,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.298538+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:45:21.298538+0000","last_peered":"2024-09-11T14:45:21.298538+0000","last_clean":"2024-09-11T14:45:21.298538+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:45:21.298538+0000","last_undegraded":"2024-09-11T14:45:21.298538+0000","last_fullsized":"2024-09-11T14:45:21.298538+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.358391+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:45:21.358391+0000","last_peered":"2024-09-11T14:45:21.358391+0000","last_clean":"2024-09-11T14:45:21.358391+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:45:21.358391+0000","last_undegraded":"2024-09-11T14:45:21.358391+0000","last_fullsized":"2024-09-11T14:45:21.358391+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"31'1399","reported_seq":1447,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:39.550296+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:45:39.550296+0000","last_peered":"2024-09-11T14:45:39.550296+0000","last_clean":"2024-09-11T14:45:39.550296+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:45:39.550296+0000","last_undegraded":"2024-09-11T14:45:39.550296+0000","last_fullsized":"2024-09-11T14:45:39.550296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1399,"log_dups_size":0,"ondisk_log_size":1399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14614528,"num_objects":223,"num_object_clones":0,"num_object_copies":669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1399,"num_write_kb":51904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.302556+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:45:21.302556+0000","last_peered":"2024-09-11T14:45:21.302556+0000","last_clean":"2024-09-11T14:45:21.302556+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:45:21.302556+0000","last_undegraded":"2024-09-11T14:45:21.302556+0000","last_fullsized":"2024-09-11T14:45:21.302556+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"31'1396","reported_seq":1444,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:39.806392+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:45:39.806392+0000","last_peered":"2024-09-11T14:45:39.806392+0000","last_clean":"2024-09-11T14:45:39.806392+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:45:39.806392+0000","last_undegraded":"2024-09-11T14:45:39.806392+0000","last_fullsized":"2024-09-11T14:45:39.806392+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1396,"log_dups_size":0,"ondisk_log_size":1396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15335424,"num_objects":234,"num_object_clones":0,"num_object_copies":702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1396,"num_write_kb":52160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.354421+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:45:21.354421+0000","last_peered":"2024-09-11T14:45:21.354421+0000","last_clean":"2024-09-11T14:45:21.354421+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:45:21.354421+0000","last_undegraded":"2024-09-11T14:45:21.354421+0000","last_fullsized":"2024-09-11T14:45:21.354421+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"31'1408","reported_seq":1456,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:38.861719+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:45:38.861719+0000","last_peered":"2024-09-11T14:45:38.861719+0000","last_clean":"2024-09-11T14:45:38.861719+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:45:38.861719+0000","last_undegraded":"2024-09-11T14:45:38.861719+0000","last_fullsized":"2024-09-11T14:45:38.861719+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1408,"log_dups_size":0,"ondisk_log_size":1408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14286848,"num_objects":218,"num_object_clones":0,"num_object_copies":654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1408,"num_write_kb":52032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.303454+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:45:21.303454+0000","last_peered":"2024-09-11T14:45:21.303454+0000","last_clean":"2024-09-11T14:45:21.303454+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:45:21.303454+0000","last_undegraded":"2024-09-11T14:45:21.303454+0000","last_fullsized":"2024-09-11T14:45:21.303454+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"31'1357","reported_seq":1405,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:35.282987+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:45:35.282987+0000","last_peered":"2024-09-11T14:45:35.282987+0000","last_clean":"2024-09-11T14:45:35.282987+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:45:35.282987+0000","last_undegraded":"2024-09-11T14:45:35.282987+0000","last_fullsized":"2024-09-11T14:45:35.282987+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1357,"log_dups_size":0,"ondisk_log_size":1357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15794176,"num_objects":241,"num_object_clones":0,"num_object_copies":723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1357,"num_write_kb":51136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"31'1432","reported_seq":1480,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:39.542889+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:45:39.542889+0000","last_peered":"2024-09-11T14:45:39.542889+0000","last_clean":"2024-09-11T14:45:39.542889+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:45:39.542889+0000","last_undegraded":"2024-09-11T14:45:39.542889+0000","last_fullsized":"2024-09-11T14:45:39.542889+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1432,"log_dups_size":0,"ondisk_log_size":1432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14942208,"num_objects":228,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1432,"num_write_kb":53120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.308554+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:45:21.308554+0000","last_peered":"2024-09-11T14:45:21.308554+0000","last_clean":"2024-09-11T14:45:21.308554+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:45:21.308554+0000","last_undegraded":"2024-09-11T14:45:21.308554+0000","last_fullsized":"2024-09-11T14:45:21.308554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"31'1309","reported_seq":1357,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:35.740956+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:45:35.740956+0000","last_peered":"2024-09-11T14:45:35.740956+0000","last_clean":"2024-09-11T14:45:35.740956+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:45:35.740956+0000","last_undegraded":"2024-09-11T14:45:35.740956+0000","last_fullsized":"2024-09-11T14:45:35.740956+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1309,"log_dups_size":0,"ondisk_log_size":1309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12779520,"num_objects":195,"num_object_clones":0,"num_object_copies":585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1309,"num_write_kb":48128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.299078+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:45:21.299078+0000","last_peered":"2024-09-11T14:45:21.299078+0000","last_clean":"2024-09-11T14:45:21.299078+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:45:21.299078+0000","last_undegraded":"2024-09-11T14:45:21.299078+0000","last_fullsized":"2024-09-11T14:45:21.299078+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"31'1437","reported_seq":1485,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:39.702864+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:45:39.702864+0000","last_peered":"2024-09-11T14:45:39.702864+0000","last_clean":"2024-09-11T14:45:39.702864+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:45:39.702864+0000","last_undegraded":"2024-09-11T14:45:39.702864+0000","last_fullsized":"2024-09-11T14:45:39.702864+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1437,"log_dups_size":0,"ondisk_log_size":1437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14221312,"num_objects":217,"num_object_clones":0,"num_object_copies":651,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1437,"num_write_kb":52928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.292230+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:45:21.292230+0000","last_peered":"2024-09-11T14:45:21.292230+0000","last_clean":"2024-09-11T14:45:21.292230+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:45:21.292230+0000","last_undegraded":"2024-09-11T14:45:21.292230+0000","last_fullsized":"2024-09-11T14:45:21.292230+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"31'1420","reported_seq":1468,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:35.530054+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:45:35.530054+0000","last_peered":"2024-09-11T14:45:35.530054+0000","last_clean":"2024-09-11T14:45:35.530054+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:45:35.530054+0000","last_undegraded":"2024-09-11T14:45:35.530054+0000","last_fullsized":"2024-09-11T14:45:35.530054+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1420,"log_dups_size":0,"ondisk_log_size":1420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13631488,"num_objects":208,"num_object_clones":0,"num_object_copies":624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1420,"num_write_kb":52096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.358988+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:45:21.358988+0000","last_peered":"2024-09-11T14:45:21.358988+0000","last_clean":"2024-09-11T14:45:21.358988+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:45:21.358988+0000","last_undegraded":"2024-09-11T14:45:21.358988+0000","last_fullsized":"2024-09-11T14:45:21.358988+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":228982808,"num_objects":3495,"num_object_clones":0,"num_object_copies":10485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22713,"num_write_kb":838593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":212784308224,"internally_reserved":0,"allocated":1427185664,"data_stored":1427185664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22713,"ondisk_log_size":22713,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":212784308224,"internally_reserved":0,"allocated":1427185664,"data_stored":1427185664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53147885568,"internally_reserved":0,"allocated":404987904,"data_stored":404987904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673229,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":341460,"kb_used_data":341460,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51956268,"statfs":{"total":53552873472,"available":53203218432,"internally_reserved":0,"allocated":349655040,"data_stored":349655040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705938,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":401564,"kb_used_data":401564,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51896164,"statfs":{"total":53552873472,"available":53141671936,"internally_reserved":0,"allocated":411201536,"data_stored":411201536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705938,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":353576,"kb_used_data":353576,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51944152,"statfs":{"total":53552873472,"available":53190811648,"internally_reserved":0,"allocated":362061824,"data_stored":362061824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705938,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":317076,"kb_used_data":317076,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51980652,"statfs":{"total":53552873472,"available":53228187648,"internally_reserved":0,"allocated":324685824,"data_stored":324685824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53147885568,"internally_reserved":0,"allocated":404987904,"data_stored":404987904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53232386048,"internally_reserved":0,"allocated":320487424,"data_stored":320487424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53196308480,"internally_reserved":0,"allocated":356564992,"data_stored":356564992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53207728128,"internally_reserved":0,"allocated":345145344,"data_stored":345145344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53147885568,"internally_reserved":0,"allocated":404987904,"data_stored":404987904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":53232386048,"internally_reserved":0,"allocated":320487424,"data_stored":320487424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":53196308480,"internally_reserved":0,"allocated":356564992,"data_stored":356564992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":53207728128,"internally_reserved":0,"allocated":345145344,"data_stored":345145344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":53147885568,"internally_reserved":0,"allocated":404987904,"data_stored":404987904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:45:41.210 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:45:41.337 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:41.333+0000 7f813ccfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:41.341 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:41.340+0000 7f813ccfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:43.963 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:45:44.136 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:44.130+0000 7fb317591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:44.143 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:44.140+0000 7fb317591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:45.913 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:45:45.913 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:45:45.913 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-11T14:45:45.914 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-11T14:45:46.060 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:46.048+0000 7f3ba4a9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:46.067 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:46.066+0000 7f3ba4a9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:46.131 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:45:46.131 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:45:46.132 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:45:46.132 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:45:46.132 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:45:46.132 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:45:46.132 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:45:46.132 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:45:46.132 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:45:46.132 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:45:46.132 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:45:46.133 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:45:46.135 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:45:46.136 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:45:46.136 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-11T14:45:46.277 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:46.267+0000 7fbb6fa88640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:46.282 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:46.281+0000 7fbb6fa88640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:46.300 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:45:46.300 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:45:46.300 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:45:46.301 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:45:46.301 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:45:46.301 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:45:46.301 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:45:46.301 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:45:46.301 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:45:46.301 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:45:46.301 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:45:46.301 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:45:46.304 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:45:46.304 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:45:46.304 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-11T14:45:46.429 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:46.417+0000 7f015a8a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:46.434 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:46.432+0000 7f015a8a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:46.582 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:45:46.582 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:45:46.582 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:45:46.582 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:45:46.582 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:45:46.582 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:45:46.583 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:45:46.583 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:45:46.583 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:45:46.583 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:45:46.583 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:45:46.583 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:45:46.585 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:45:46.585 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:45:46.585 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-11T14:45:46.717 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:46.709+0000 7fb155d51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:46.722 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:46.720+0000 7fb155d51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:46.723 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:45:46.839 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:46.836+0000 7f0deb85a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:46.843 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:46.842+0000 7f0deb85a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:46.911 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:45:46.911 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:45:46.911 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:45:46.911 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:45:46.911 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:45:46.911 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:45:46.912 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:45:46.912 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:45:46.912 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:45:46.912 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:45:46.912 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:45:46.912 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:45:46.914 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:45:46.914 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:45:49.333 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:45:49.451 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:49.448+0000 7f03e20d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:49.455 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:49.454+0000 7f03e20d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:51.915 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:45:51.915 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:45:51.916 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:45:52.054 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:52.045+0000 7fb858391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:52.059 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:52.058+0000 7fb858391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:52.092 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:45:52.233 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:52.229+0000 7f571ef51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:52.239 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:52.237+0000 7f571ef51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:52.317 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:45:52.318 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:45:52.337 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-09-11T14:45:50.769376+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":188678731,"num_objects":2875,"num_object_clones":0,"num_object_copies":8625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":23418,"num_write_kb":839701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23392,"ondisk_log_size":23392,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":1172252,"kb_used_data":1172252,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":208018660,"statfs":{"total":214211493888,"available":213011107840,"internally_reserved":0,"allocated":1200386048,"data_stored":1200386048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-60489728,"num_objects":-923,"num_object_clones":0,"num_object_copies":-2769,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":923,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002658"},"pg_stats":[{"pgid":"3.f","version":"31'1439","reported_seq":1487,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:49.344305+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:45:49.344305+0000","last_peered":"2024-09-11T14:45:49.344305+0000","last_clean":"2024-09-11T14:45:49.344305+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:45:49.344305+0000","last_undegraded":"2024-09-11T14:45:49.344305+0000","last_fullsized":"2024-09-11T14:45:49.344305+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1439,"log_dups_size":0,"ondisk_log_size":1439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11730944,"num_objects":179,"num_object_clones":0,"num_object_copies":537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1439,"num_write_kb":51776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"31'1580","reported_seq":1628,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:45.250526+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:45:45.250526+0000","last_peered":"2024-09-11T14:45:45.250526+0000","last_clean":"2024-09-11T14:45:45.250526+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:45:45.250526+0000","last_undegraded":"2024-09-11T14:45:45.250526+0000","last_fullsized":"2024-09-11T14:45:45.250526+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1580,"log_dups_size":0,"ondisk_log_size":1580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13107200,"num_objects":200,"num_object_clones":0,"num_object_copies":600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1580,"num_write_kb":56960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"31'1487","reported_seq":1536,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:49.700547+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:45:49.700547+0000","last_peered":"2024-09-11T14:45:49.700547+0000","last_clean":"2024-09-11T14:45:49.700547+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:45:49.700547+0000","last_undegraded":"2024-09-11T14:45:49.700547+0000","last_fullsized":"2024-09-11T14:45:49.700547+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1487,"log_dups_size":0,"ondisk_log_size":1487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11796504,"num_objects":181,"num_object_clones":0,"num_object_copies":543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1487,"num_write_kb":53313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"31'1424","reported_seq":1472,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:47.962004+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:45:47.962004+0000","last_peered":"2024-09-11T14:45:47.962004+0000","last_clean":"2024-09-11T14:45:47.962004+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:45:47.962004+0000","last_undegraded":"2024-09-11T14:45:47.962004+0000","last_fullsized":"2024-09-11T14:45:47.962004+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1424,"log_dups_size":0,"ondisk_log_size":1424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10223616,"num_objects":156,"num_object_clones":0,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1424,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"31'1427","reported_seq":1475,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:49.641741+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:45:49.641741+0000","last_peered":"2024-09-11T14:45:49.641741+0000","last_clean":"2024-09-11T14:45:49.641741+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:45:49.641741+0000","last_undegraded":"2024-09-11T14:45:49.641741+0000","last_fullsized":"2024-09-11T14:45:49.641741+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1427,"log_dups_size":0,"ondisk_log_size":1427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11599872,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1427,"num_write_kb":51328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"31'1581","reported_seq":1629,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:49.751646+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:45:49.751646+0000","last_peered":"2024-09-11T14:45:49.751646+0000","last_clean":"2024-09-11T14:45:49.751646+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:45:49.751646+0000","last_undegraded":"2024-09-11T14:45:49.751646+0000","last_fullsized":"2024-09-11T14:45:49.751646+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1581,"log_dups_size":0,"ondisk_log_size":1581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12255232,"num_objects":187,"num_object_clones":0,"num_object_copies":561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1581,"num_write_kb":56576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"31'1505","reported_seq":1553,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:49.165327+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:45:49.165327+0000","last_peered":"2024-09-11T14:45:49.165327+0000","last_clean":"2024-09-11T14:45:49.165327+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:45:49.165327+0000","last_undegraded":"2024-09-11T14:45:49.165327+0000","last_fullsized":"2024-09-11T14:45:49.165327+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1505,"log_dups_size":0,"ondisk_log_size":1505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11075584,"num_objects":169,"num_object_clones":0,"num_object_copies":507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1505,"num_write_kb":53568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"31'1428","reported_seq":1476,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:49.632919+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:45:49.632919+0000","last_peered":"2024-09-11T14:45:49.632919+0000","last_clean":"2024-09-11T14:45:49.632919+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:45:49.632919+0000","last_undegraded":"2024-09-11T14:45:49.632919+0000","last_fullsized":"2024-09-11T14:45:49.632919+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1428,"log_dups_size":0,"ondisk_log_size":1428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10878976,"num_objects":166,"num_object_clones":0,"num_object_copies":498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1428,"num_write_kb":51008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":133,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.298538+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:45:21.298538+0000","last_peered":"2024-09-11T14:45:21.298538+0000","last_clean":"2024-09-11T14:45:21.298538+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:45:21.298538+0000","last_undegraded":"2024-09-11T14:45:21.298538+0000","last_fullsized":"2024-09-11T14:45:21.298538+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.358391+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:45:21.358391+0000","last_peered":"2024-09-11T14:45:21.358391+0000","last_clean":"2024-09-11T14:45:21.358391+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:45:21.358391+0000","last_undegraded":"2024-09-11T14:45:21.358391+0000","last_fullsized":"2024-09-11T14:45:21.358391+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"31'1430","reported_seq":1478,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:49.616487+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:45:49.616487+0000","last_peered":"2024-09-11T14:45:49.616487+0000","last_clean":"2024-09-11T14:45:49.616487+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:45:49.616487+0000","last_undegraded":"2024-09-11T14:45:49.616487+0000","last_fullsized":"2024-09-11T14:45:49.616487+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1430,"log_dups_size":0,"ondisk_log_size":1430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1430,"num_write_kb":51904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.302556+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:45:21.302556+0000","last_peered":"2024-09-11T14:45:21.302556+0000","last_clean":"2024-09-11T14:45:21.302556+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:45:21.302556+0000","last_undegraded":"2024-09-11T14:45:21.302556+0000","last_fullsized":"2024-09-11T14:45:21.302556+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"31'1435","reported_seq":1483,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:49.029266+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:45:49.029266+0000","last_peered":"2024-09-11T14:45:49.029266+0000","last_clean":"2024-09-11T14:45:49.029266+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:45:49.029266+0000","last_undegraded":"2024-09-11T14:45:49.029266+0000","last_fullsized":"2024-09-11T14:45:49.029266+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1435,"log_dups_size":0,"ondisk_log_size":1435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12779520,"num_objects":195,"num_object_clones":0,"num_object_copies":585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1435,"num_write_kb":52160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.354421+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:45:21.354421+0000","last_peered":"2024-09-11T14:45:21.354421+0000","last_clean":"2024-09-11T14:45:21.354421+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:45:21.354421+0000","last_undegraded":"2024-09-11T14:45:21.354421+0000","last_fullsized":"2024-09-11T14:45:21.354421+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"31'1453","reported_seq":1501,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:49.103236+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:45:49.103236+0000","last_peered":"2024-09-11T14:45:49.103236+0000","last_clean":"2024-09-11T14:45:49.103236+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:45:49.103236+0000","last_undegraded":"2024-09-11T14:45:49.103236+0000","last_fullsized":"2024-09-11T14:45:49.103236+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1453,"log_dups_size":0,"ondisk_log_size":1453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11337728,"num_objects":173,"num_object_clones":0,"num_object_copies":519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1453,"num_write_kb":52032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.303454+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:45:21.303454+0000","last_peered":"2024-09-11T14:45:21.303454+0000","last_clean":"2024-09-11T14:45:21.303454+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:45:21.303454+0000","last_undegraded":"2024-09-11T14:45:21.303454+0000","last_fullsized":"2024-09-11T14:45:21.303454+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"31'1403","reported_seq":1451,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:45.478459+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:45:45.478459+0000","last_peered":"2024-09-11T14:45:45.478459+0000","last_clean":"2024-09-11T14:45:45.478459+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:45:45.478459+0000","last_undegraded":"2024-09-11T14:45:45.478459+0000","last_fullsized":"2024-09-11T14:45:45.478459+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1403,"log_dups_size":0,"ondisk_log_size":1403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12779520,"num_objects":195,"num_object_clones":0,"num_object_copies":585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1403,"num_write_kb":51136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"31'1478","reported_seq":1526,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:49.754980+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:45:49.754980+0000","last_peered":"2024-09-11T14:45:49.754980+0000","last_clean":"2024-09-11T14:45:49.754980+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:45:49.754980+0000","last_undegraded":"2024-09-11T14:45:49.754980+0000","last_fullsized":"2024-09-11T14:45:49.754980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1478,"log_dups_size":0,"ondisk_log_size":1478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11927552,"num_objects":182,"num_object_clones":0,"num_object_copies":546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1478,"num_write_kb":53120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.308554+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:45:21.308554+0000","last_peered":"2024-09-11T14:45:21.308554+0000","last_clean":"2024-09-11T14:45:21.308554+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:45:21.308554+0000","last_undegraded":"2024-09-11T14:45:21.308554+0000","last_fullsized":"2024-09-11T14:45:21.308554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"31'1339","reported_seq":1387,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:44.472029+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:45:44.472029+0000","last_peered":"2024-09-11T14:45:44.472029+0000","last_clean":"2024-09-11T14:45:44.472029+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:45:44.472029+0000","last_undegraded":"2024-09-11T14:45:44.472029+0000","last_fullsized":"2024-09-11T14:45:44.472029+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1339,"log_dups_size":0,"ondisk_log_size":1339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10813440,"num_objects":165,"num_object_clones":0,"num_object_copies":495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1339,"num_write_kb":48128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.299078+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:45:21.299078+0000","last_peered":"2024-09-11T14:45:21.299078+0000","last_clean":"2024-09-11T14:45:21.299078+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:45:21.299078+0000","last_undegraded":"2024-09-11T14:45:21.299078+0000","last_fullsized":"2024-09-11T14:45:21.299078+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"31'1466","reported_seq":1514,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:49.024950+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:45:49.024950+0000","last_peered":"2024-09-11T14:45:49.024950+0000","last_clean":"2024-09-11T14:45:49.024950+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:45:49.024950+0000","last_undegraded":"2024-09-11T14:45:49.024950+0000","last_fullsized":"2024-09-11T14:45:49.024950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1466,"log_dups_size":0,"ondisk_log_size":1466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12320768,"num_objects":188,"num_object_clones":0,"num_object_copies":564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1466,"num_write_kb":52928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.292230+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:45:21.292230+0000","last_peered":"2024-09-11T14:45:21.292230+0000","last_clean":"2024-09-11T14:45:21.292230+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:45:21.292230+0000","last_undegraded":"2024-09-11T14:45:21.292230+0000","last_fullsized":"2024-09-11T14:45:21.292230+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"31'1462","reported_seq":1510,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:45.395664+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:45:45.395664+0000","last_peered":"2024-09-11T14:45:45.395664+0000","last_clean":"2024-09-11T14:45:45.395664+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:45:45.395664+0000","last_undegraded":"2024-09-11T14:45:45.395664+0000","last_fullsized":"2024-09-11T14:45:45.395664+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1462,"log_dups_size":0,"ondisk_log_size":1462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10878976,"num_objects":166,"num_object_clones":0,"num_object_copies":498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1462,"num_write_kb":52096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.358988+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:45:21.358988+0000","last_peered":"2024-09-11T14:45:21.358988+0000","last_clean":"2024-09-11T14:45:21.358988+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:45:21.358988+0000","last_undegraded":"2024-09-11T14:45:21.358988+0000","last_fullsized":"2024-09-11T14:45:21.358988+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":188088344,"num_objects":2871,"num_object_clones":0,"num_object_copies":8613,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":23337,"num_write_kb":838593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":213027053568,"internally_reserved":0,"allocated":1184440320,"data_stored":1184440320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23337,"ondisk_log_size":23337,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":213026791424,"internally_reserved":0,"allocated":1184702464,"data_stored":1184702464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53214384128,"internally_reserved":0,"allocated":338489344,"data_stored":338489344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673239,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":277880,"kb_used_data":277880,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52019848,"statfs":{"total":53552873472,"available":53268324352,"internally_reserved":0,"allocated":284549120,"data_stored":284549120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705948,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":335692,"kb_used_data":335692,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51962036,"statfs":{"total":53552873472,"available":53209124864,"internally_reserved":0,"allocated":343748608,"data_stored":343748608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705948,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":292508,"kb_used_data":292508,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52005220,"statfs":{"total":53552873472,"available":53253345280,"internally_reserved":0,"allocated":299528192,"data_stored":299528192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705948,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":266172,"kb_used_data":266172,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52031556,"statfs":{"total":53552873472,"available":53280313344,"internally_reserved":0,"allocated":272560128,"data_stored":272560128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53214384128,"internally_reserved":0,"allocated":338489344,"data_stored":338489344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53283090432,"internally_reserved":0,"allocated":269783040,"data_stored":269783040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53257953280,"internally_reserved":0,"allocated":294920192,"data_stored":294920192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53271363584,"internally_reserved":0,"allocated":281509888,"data_stored":281509888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53214384128,"internally_reserved":0,"allocated":338489344,"data_stored":338489344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":53283221504,"internally_reserved":0,"allocated":269651968,"data_stored":269651968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":53257953280,"internally_reserved":0,"allocated":294920192,"data_stored":294920192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":53271494656,"internally_reserved":0,"allocated":281378816,"data_stored":281378816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":53214384128,"internally_reserved":0,"allocated":338489344,"data_stored":338489344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:45:54.826 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:45:54.996 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:54.992+0000 7fe0dc27f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:55.000 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:54.999+0000 7fe0dc27f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:57.339 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:45:57.339 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:45:57.340 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:45:57.460 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:57.454+0000 7f710c874640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:57.465 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:57.464+0000 7f710c874640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:57.733 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:45:57.747 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:45:57.748 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:45:57.770 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":170,"stamp":"2024-09-11T14:45:56.770581+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":165937739,"num_objects":2528,"num_object_clones":0,"num_object_copies":7584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":23765,"num_write_kb":839701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23739,"ondisk_log_size":23739,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":1039480,"kb_used_data":1039480,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":208151432,"statfs":{"total":214211493888,"available":213147066368,"internally_reserved":0,"allocated":1064427520,"data_stored":1064427520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-57212928,"num_objects":-873,"num_object_clones":0,"num_object_copies":-2619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":873,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002422"},"pg_stats":[{"pgid":"3.f","version":"31'1452","reported_seq":1500,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:54.343616+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:45:54.343616+0000","last_peered":"2024-09-11T14:45:54.343616+0000","last_clean":"2024-09-11T14:45:54.343616+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:45:54.343616+0000","last_undegraded":"2024-09-11T14:45:54.343616+0000","last_fullsized":"2024-09-11T14:45:54.343616+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1452,"log_dups_size":0,"ondisk_log_size":1452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10878976,"num_objects":166,"num_object_clones":0,"num_object_copies":498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1452,"num_write_kb":51776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"31'1624","reported_seq":1672,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:55.538589+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:45:55.538589+0000","last_peered":"2024-09-11T14:45:55.538589+0000","last_clean":"2024-09-11T14:45:55.538589+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:45:55.538589+0000","last_undegraded":"2024-09-11T14:45:55.538589+0000","last_fullsized":"2024-09-11T14:45:55.538589+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1624,"log_dups_size":0,"ondisk_log_size":1624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10223616,"num_objects":156,"num_object_clones":0,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1624,"num_write_kb":56960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"31'1506","reported_seq":1555,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:54.485611+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:45:54.485611+0000","last_peered":"2024-09-11T14:45:54.485611+0000","last_clean":"2024-09-11T14:45:54.485611+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:45:54.485611+0000","last_undegraded":"2024-09-11T14:45:54.485611+0000","last_fullsized":"2024-09-11T14:45:54.485611+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1506,"log_dups_size":0,"ondisk_log_size":1506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10551320,"num_objects":162,"num_object_clones":0,"num_object_copies":486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1506,"num_write_kb":53313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"31'1441","reported_seq":1489,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:54.757111+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:45:54.757111+0000","last_peered":"2024-09-11T14:45:54.757111+0000","last_clean":"2024-09-11T14:45:54.757111+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:45:54.757111+0000","last_undegraded":"2024-09-11T14:45:54.757111+0000","last_fullsized":"2024-09-11T14:45:54.757111+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1441,"log_dups_size":0,"ondisk_log_size":1441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9109504,"num_objects":139,"num_object_clones":0,"num_object_copies":417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1441,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"31'1444","reported_seq":1492,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:54.227151+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:45:54.227151+0000","last_peered":"2024-09-11T14:45:54.227151+0000","last_clean":"2024-09-11T14:45:54.227151+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:45:54.227151+0000","last_undegraded":"2024-09-11T14:45:54.227151+0000","last_fullsized":"2024-09-11T14:45:54.227151+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1444,"log_dups_size":0,"ondisk_log_size":1444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10485760,"num_objects":160,"num_object_clones":0,"num_object_copies":480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1444,"num_write_kb":51328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"31'1599","reported_seq":1647,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:54.725833+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:45:54.725833+0000","last_peered":"2024-09-11T14:45:54.725833+0000","last_clean":"2024-09-11T14:45:54.725833+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:45:54.725833+0000","last_undegraded":"2024-09-11T14:45:54.725833+0000","last_fullsized":"2024-09-11T14:45:54.725833+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1599,"log_dups_size":0,"ondisk_log_size":1599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11075584,"num_objects":169,"num_object_clones":0,"num_object_copies":507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1599,"num_write_kb":56576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"31'1521","reported_seq":1569,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:54.242282+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:45:54.242282+0000","last_peered":"2024-09-11T14:45:54.242282+0000","last_clean":"2024-09-11T14:45:54.242282+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:45:54.242282+0000","last_undegraded":"2024-09-11T14:45:54.242282+0000","last_fullsized":"2024-09-11T14:45:54.242282+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1521,"log_dups_size":0,"ondisk_log_size":1521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10027008,"num_objects":153,"num_object_clones":0,"num_object_copies":459,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1521,"num_write_kb":53568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"31'1445","reported_seq":1493,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:54.801019+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:45:54.801019+0000","last_peered":"2024-09-11T14:45:54.801019+0000","last_clean":"2024-09-11T14:45:54.801019+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:45:54.801019+0000","last_undegraded":"2024-09-11T14:45:54.801019+0000","last_fullsized":"2024-09-11T14:45:54.801019+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1445,"log_dups_size":0,"ondisk_log_size":1445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9764864,"num_objects":149,"num_object_clones":0,"num_object_copies":447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1445,"num_write_kb":51008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":133,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.298538+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:45:21.298538+0000","last_peered":"2024-09-11T14:45:21.298538+0000","last_clean":"2024-09-11T14:45:21.298538+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:45:21.298538+0000","last_undegraded":"2024-09-11T14:45:21.298538+0000","last_fullsized":"2024-09-11T14:45:21.298538+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.358391+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:45:21.358391+0000","last_peered":"2024-09-11T14:45:21.358391+0000","last_clean":"2024-09-11T14:45:21.358391+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:45:21.358391+0000","last_undegraded":"2024-09-11T14:45:21.358391+0000","last_fullsized":"2024-09-11T14:45:21.358391+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"31'1447","reported_seq":1495,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:54.699473+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:45:54.699473+0000","last_peered":"2024-09-11T14:45:54.699473+0000","last_clean":"2024-09-11T14:45:54.699473+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:45:54.699473+0000","last_undegraded":"2024-09-11T14:45:54.699473+0000","last_fullsized":"2024-09-11T14:45:54.699473+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1447,"log_dups_size":0,"ondisk_log_size":1447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11468800,"num_objects":175,"num_object_clones":0,"num_object_copies":525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1447,"num_write_kb":51904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.302556+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:45:21.302556+0000","last_peered":"2024-09-11T14:45:21.302556+0000","last_clean":"2024-09-11T14:45:21.302556+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:45:21.302556+0000","last_undegraded":"2024-09-11T14:45:21.302556+0000","last_fullsized":"2024-09-11T14:45:21.302556+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"31'1457","reported_seq":1505,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:54.565226+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:45:54.565226+0000","last_peered":"2024-09-11T14:45:54.565226+0000","last_clean":"2024-09-11T14:45:54.565226+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:45:54.565226+0000","last_undegraded":"2024-09-11T14:45:54.565226+0000","last_fullsized":"2024-09-11T14:45:54.565226+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1457,"log_dups_size":0,"ondisk_log_size":1457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11337728,"num_objects":173,"num_object_clones":0,"num_object_copies":519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1457,"num_write_kb":52160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.354421+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:45:21.354421+0000","last_peered":"2024-09-11T14:45:21.354421+0000","last_clean":"2024-09-11T14:45:21.354421+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:45:21.354421+0000","last_undegraded":"2024-09-11T14:45:21.354421+0000","last_fullsized":"2024-09-11T14:45:21.354421+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"31'1463","reported_seq":1511,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:54.314375+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:45:54.314375+0000","last_peered":"2024-09-11T14:45:54.314375+0000","last_clean":"2024-09-11T14:45:54.314375+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:45:54.314375+0000","last_undegraded":"2024-09-11T14:45:54.314375+0000","last_fullsized":"2024-09-11T14:45:54.314375+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1463,"log_dups_size":0,"ondisk_log_size":1463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10682368,"num_objects":163,"num_object_clones":0,"num_object_copies":489,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1463,"num_write_kb":52032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.303454+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:45:21.303454+0000","last_peered":"2024-09-11T14:45:21.303454+0000","last_clean":"2024-09-11T14:45:21.303454+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:45:21.303454+0000","last_undegraded":"2024-09-11T14:45:21.303454+0000","last_fullsized":"2024-09-11T14:45:21.303454+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"31'1435","reported_seq":1483,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:55.135354+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:45:55.135354+0000","last_peered":"2024-09-11T14:45:55.135354+0000","last_clean":"2024-09-11T14:45:55.135354+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:45:55.135354+0000","last_undegraded":"2024-09-11T14:45:55.135354+0000","last_fullsized":"2024-09-11T14:45:55.135354+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1435,"log_dups_size":0,"ondisk_log_size":1435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10682368,"num_objects":163,"num_object_clones":0,"num_object_copies":489,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1435,"num_write_kb":51136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"31'1500","reported_seq":1548,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:54.394409+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:45:54.394409+0000","last_peered":"2024-09-11T14:45:54.394409+0000","last_clean":"2024-09-11T14:45:54.394409+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:45:54.394409+0000","last_undegraded":"2024-09-11T14:45:54.394409+0000","last_fullsized":"2024-09-11T14:45:54.394409+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1500,"log_dups_size":0,"ondisk_log_size":1500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10485760,"num_objects":160,"num_object_clones":0,"num_object_copies":480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1500,"num_write_kb":53120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.308554+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:45:21.308554+0000","last_peered":"2024-09-11T14:45:21.308554+0000","last_clean":"2024-09-11T14:45:21.308554+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:45:21.308554+0000","last_undegraded":"2024-09-11T14:45:21.308554+0000","last_fullsized":"2024-09-11T14:45:21.308554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"31'1376","reported_seq":1424,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:55.190554+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:45:55.190554+0000","last_peered":"2024-09-11T14:45:55.190554+0000","last_clean":"2024-09-11T14:45:55.190554+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:45:55.190554+0000","last_undegraded":"2024-09-11T14:45:55.190554+0000","last_fullsized":"2024-09-11T14:45:55.190554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1376,"log_dups_size":0,"ondisk_log_size":1376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1376,"num_write_kb":48128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.299078+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:45:21.299078+0000","last_peered":"2024-09-11T14:45:21.299078+0000","last_clean":"2024-09-11T14:45:21.299078+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:45:21.299078+0000","last_undegraded":"2024-09-11T14:45:21.299078+0000","last_fullsized":"2024-09-11T14:45:21.299078+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"31'1484","reported_seq":1532,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:54.567889+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:45:54.567889+0000","last_peered":"2024-09-11T14:45:54.567889+0000","last_clean":"2024-09-11T14:45:54.567889+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:45:54.567889+0000","last_undegraded":"2024-09-11T14:45:54.567889+0000","last_fullsized":"2024-09-11T14:45:54.567889+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1484,"log_dups_size":0,"ondisk_log_size":1484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11141120,"num_objects":170,"num_object_clones":0,"num_object_copies":510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1484,"num_write_kb":52928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.292230+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:45:21.292230+0000","last_peered":"2024-09-11T14:45:21.292230+0000","last_clean":"2024-09-11T14:45:21.292230+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:45:21.292230+0000","last_undegraded":"2024-09-11T14:45:21.292230+0000","last_fullsized":"2024-09-11T14:45:21.292230+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"31'1490","reported_seq":1538,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:55.246449+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:45:55.246449+0000","last_peered":"2024-09-11T14:45:55.246449+0000","last_clean":"2024-09-11T14:45:55.246449+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:45:55.246449+0000","last_undegraded":"2024-09-11T14:45:55.246449+0000","last_fullsized":"2024-09-11T14:45:55.246449+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1490,"log_dups_size":0,"ondisk_log_size":1490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9043968,"num_objects":138,"num_object_clones":0,"num_object_copies":414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1490,"num_write_kb":52096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.358988+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:45:21.358988+0000","last_peered":"2024-09-11T14:45:21.358988+0000","last_clean":"2024-09-11T14:45:21.358988+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:45:21.358988+0000","last_undegraded":"2024-09-11T14:45:21.358988+0000","last_fullsized":"2024-09-11T14:45:21.358988+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":165347352,"num_objects":2524,"num_object_clones":0,"num_object_copies":7572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":23684,"num_write_kb":838593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":213163847680,"internally_reserved":0,"allocated":1047646208,"data_stored":1047646208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23684,"ondisk_log_size":23684,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":213163716608,"internally_reserved":0,"allocated":1047777280,"data_stored":1047777280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53246058496,"internally_reserved":0,"allocated":306814976,"data_stored":306814976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673249,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":227544,"kb_used_data":227544,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52070184,"statfs":{"total":53552873472,"available":53319868416,"internally_reserved":0,"allocated":233005056,"data_stored":233005056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705953,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":304780,"kb_used_data":304780,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":51992948,"statfs":{"total":53552873472,"available":53240778752,"internally_reserved":0,"allocated":312094720,"data_stored":312094720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705953,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":265984,"kb_used_data":265984,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52031744,"statfs":{"total":53552873472,"available":53280505856,"internally_reserved":0,"allocated":272367616,"data_stored":272367616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705953,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":241172,"kb_used_data":241172,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52056556,"statfs":{"total":53552873472,"available":53305913344,"internally_reserved":0,"allocated":246960128,"data_stored":246960128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53246058496,"internally_reserved":0,"allocated":306814976,"data_stored":306814976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53308964864,"internally_reserved":0,"allocated":243908608,"data_stored":243908608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53284868096,"internally_reserved":0,"allocated":268005376,"data_stored":268005376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53323563008,"internally_reserved":0,"allocated":229310464,"data_stored":229310464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53246320640,"internally_reserved":0,"allocated":306552832,"data_stored":306552832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":53308964864,"internally_reserved":0,"allocated":243908608,"data_stored":243908608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":53284999168,"internally_reserved":0,"allocated":267874304,"data_stored":267874304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":53323563008,"internally_reserved":0,"allocated":229310464,"data_stored":229310464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":53246320640,"internally_reserved":0,"allocated":306552832,"data_stored":306552832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:45:57.887 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:57.883+0000 7f5a9ac9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:57.892 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:45:57.890+0000 7f5a9ac9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:00.656 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:46:00.796 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:00.793+0000 7f5810cd1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:00.800 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:00.799+0000 7f5810cd1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:02.772 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:46:02.773 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:46:02.773 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-11T14:46:02.773 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-11T14:46:02.908 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:02.896+0000 7fdec8191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:02.913 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:02.912+0000 7fdec8191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:02.942 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:46:02.942 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:46:02.942 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:46:02.942 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:46:02.942 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:46:02.942 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:46:02.942 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:46:02.942 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:46:02.943 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:46:02.943 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:46:02.943 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:46:02.943 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:46:02.945 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:46:02.945 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:46:02.946 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-11T14:46:03.073 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:03.065+0000 7f089ab51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:03.077 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:03.076+0000 7f089ab51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:03.096 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:46:03.096 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:46:03.096 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:46:03.096 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:46:03.096 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:46:03.096 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:46:03.096 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:46:03.096 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:46:03.096 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:46:03.096 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:46:03.096 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:46:03.097 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:46:03.099 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:46:03.099 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:46:03.099 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-11T14:46:03.229 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:03.221+0000 7fa746748640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:03.233 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:03.232+0000 7fa746748640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:03.379 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:46:03.380 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:46:03.380 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:46:03.380 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:46:03.380 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:46:03.380 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:46:03.380 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:46:03.380 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:46:03.380 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:46:03.380 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:46:03.380 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:46:03.380 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:46:03.383 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:46:03.383 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:46:03.383 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-11T14:46:03.448 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:46:03.536 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:03.524+0000 7f62b78fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:03.541 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:03.539+0000 7f62b78fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:03.584 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:03.581+0000 7fd06ce5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:03.589 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:03.588+0000 7fd06ce5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:03.782 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:46:03.782 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:46:03.782 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:46:03.782 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:46:03.783 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:46:03.783 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:46:03.783 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:46:03.783 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:46:03.783 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:46:03.783 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:46:03.783 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:46:03.783 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:46:03.785 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:46:03.785 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:46:06.158 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:46:06.318 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:06.314+0000 7f99eb191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:06.323 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:06.322+0000 7f99eb191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:08.786 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:46:08.786 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:46:08.787 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-11T14:46:08.787 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-11T14:46:08.924 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:46:08.928 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:08.918+0000 7f6dda4d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:08.936 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:08.935+0000 7f6dda4d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:09.008 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:46:09.009 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:46:09.009 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:46:09.009 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:46:09.009 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:46:09.009 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:46:09.009 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:46:09.009 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:46:09.009 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:46:09.009 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:46:09.009 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:46:09.009 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:46:09.013 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:46:09.013 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:46:09.013 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-11T14:46:09.070 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:09.066+0000 7faa38f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:09.078 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:09.077+0000 7faa38f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:09.217 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:09.206+0000 7efd87ebd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:09.224 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:09.223+0000 7efd87ebd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:09.309 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:46:09.309 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:46:09.309 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:46:09.309 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:46:09.309 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:46:09.309 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:46:09.309 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:46:09.309 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:46:09.309 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:46:09.309 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:46:09.309 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:46:09.309 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:46:09.314 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:46:09.314 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:46:09.314 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-11T14:46:09.521 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:09.513+0000 7f1f25391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:09.525 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:09.524+0000 7f1f25391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:09.582 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:46:09.582 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:46:09.582 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:46:09.582 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:46:09.582 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:46:09.582 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:46:09.582 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:46:09.582 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:46:09.582 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:46:09.583 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:46:09.583 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:46:09.583 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:46:09.586 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:46:09.586 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:46:09.587 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-11T14:46:09.770 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:09.762+0000 7f6d41711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:09.774 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:09.773+0000 7f6d41711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:09.942 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-11T14:46:09.942 INFO:teuthology.orchestra.run.smithi181.stderr:bench [] [] [] [] 2024-09-11T14:46:09.942 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-11T14:46:09.942 INFO:teuthology.orchestra.run.smithi181.stderr:config help 2024-09-11T14:46:09.942 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-11T14:46:09.942 INFO:teuthology.orchestra.run.smithi181.stderr:config show 2024-09-11T14:46:09.942 INFO:teuthology.orchestra.run.smithi181.stderr:deep_scrub 2024-09-11T14:46:09.942 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_ops 2024-09-11T14:46:09.943 INFO:teuthology.orchestra.run.smithi181.stderr:dump_historic_slow_ops 2024-09-11T14:46:09.943 INFO:teuthology.orchestra.run.smithi181.stderr:dump_metrics [] 2024-09-11T14:46:09.943 INFO:teuthology.orchestra.run.smithi181.stderr:dump_ops_in_flight 2024-09-11T14:46:09.943 INFO:teuthology.orchestra.run.smithi181.stderr:Error EINVAL: invalid command 2024-09-11T14:46:09.946 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T14:46:09.946 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-11T14:46:11.547 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:46:11.697 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:11.693+0000 7f0693351640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:11.702 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:11.701+0000 7f0693351640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:14.161 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:46:14.314 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:14.310+0000 7fb5dbcbd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:14.320 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:14.319+0000 7fb5dbcbd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:14.947 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:46:14.947 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:46:14.947 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:46:15.071 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:15.065+0000 7f596026c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:15.075 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:15.074+0000 7f596026c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:15.440 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:46:15.441 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:46:15.454 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":179,"stamp":"2024-09-11T14:46:14.775137+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":101450315,"num_objects":1544,"num_object_clones":0,"num_object_copies":4632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":24749,"num_write_kb":839701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24723,"ondisk_log_size":24723,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":660928,"kb_used_data":660928,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":208529984,"statfs":{"total":214211493888,"available":213534703616,"internally_reserved":0,"allocated":676790272,"data_stored":676790272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-46202880,"num_objects":-705,"num_object_clones":0,"num_object_copies":-2115,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":705,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002994"},"pg_stats":[{"pgid":"3.f","version":"31'1524","reported_seq":1572,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:09.609853+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:46:09.609853+0000","last_peered":"2024-09-11T14:46:09.609853+0000","last_clean":"2024-09-11T14:46:09.609853+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:46:09.609853+0000","last_undegraded":"2024-09-11T14:46:09.609853+0000","last_fullsized":"2024-09-11T14:46:09.609853+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1524,"log_dups_size":0,"ondisk_log_size":1524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6160384,"num_objects":94,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1524,"num_write_kb":51776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"31'1684","reported_seq":1732,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:10.588960+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:46:10.588960+0000","last_peered":"2024-09-11T14:46:10.588960+0000","last_clean":"2024-09-11T14:46:10.588960+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:46:10.588960+0000","last_undegraded":"2024-09-11T14:46:10.588960+0000","last_fullsized":"2024-09-11T14:46:10.588960+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1684,"log_dups_size":0,"ondisk_log_size":1684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6291456,"num_objects":96,"num_object_clones":0,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1684,"num_write_kb":56960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"31'1555","reported_seq":1604,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:09.164564+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:46:09.164564+0000","last_peered":"2024-09-11T14:46:09.164564+0000","last_clean":"2024-09-11T14:46:09.164564+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:46:09.164564+0000","last_undegraded":"2024-09-11T14:46:09.164564+0000","last_fullsized":"2024-09-11T14:46:09.164564+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1555,"log_dups_size":0,"ondisk_log_size":1555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7340056,"num_objects":113,"num_object_clones":0,"num_object_copies":339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1555,"num_write_kb":53313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"31'1496","reported_seq":1544,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:09.490657+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:46:09.490657+0000","last_peered":"2024-09-11T14:46:09.490657+0000","last_clean":"2024-09-11T14:46:09.490657+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:46:09.490657+0000","last_undegraded":"2024-09-11T14:46:09.490657+0000","last_fullsized":"2024-09-11T14:46:09.490657+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1496,"log_dups_size":0,"ondisk_log_size":1496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5505024,"num_objects":84,"num_object_clones":0,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1496,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"31'1495","reported_seq":1543,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:09.492162+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:46:09.492162+0000","last_peered":"2024-09-11T14:46:09.492162+0000","last_clean":"2024-09-11T14:46:09.492162+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:46:09.492162+0000","last_undegraded":"2024-09-11T14:46:09.492162+0000","last_fullsized":"2024-09-11T14:46:09.492162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1495,"log_dups_size":0,"ondisk_log_size":1495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1495,"num_write_kb":51328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"31'1668","reported_seq":1716,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:09.858190+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:46:09.858190+0000","last_peered":"2024-09-11T14:46:09.858190+0000","last_clean":"2024-09-11T14:46:09.858190+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:46:09.858190+0000","last_undegraded":"2024-09-11T14:46:09.858190+0000","last_fullsized":"2024-09-11T14:46:09.858190+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1668,"log_dups_size":0,"ondisk_log_size":1668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6553600,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1668,"num_write_kb":56576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"31'1588","reported_seq":1636,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:09.822633+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:46:09.822633+0000","last_peered":"2024-09-11T14:46:09.822633+0000","last_clean":"2024-09-11T14:46:09.822633+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:46:09.822633+0000","last_undegraded":"2024-09-11T14:46:09.822633+0000","last_fullsized":"2024-09-11T14:46:09.822633+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1588,"log_dups_size":0,"ondisk_log_size":1588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5636096,"num_objects":86,"num_object_clones":0,"num_object_copies":258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1588,"num_write_kb":53568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"31'1503","reported_seq":1551,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:09.714181+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:46:09.714181+0000","last_peered":"2024-09-11T14:46:09.714181+0000","last_clean":"2024-09-11T14:46:09.714181+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:46:09.714181+0000","last_undegraded":"2024-09-11T14:46:09.714181+0000","last_fullsized":"2024-09-11T14:46:09.714181+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1503,"log_dups_size":0,"ondisk_log_size":1503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5963776,"num_objects":91,"num_object_clones":0,"num_object_copies":273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1503,"num_write_kb":51008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":133,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.298538+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:45:21.298538+0000","last_peered":"2024-09-11T14:45:21.298538+0000","last_clean":"2024-09-11T14:45:21.298538+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:45:21.298538+0000","last_undegraded":"2024-09-11T14:45:21.298538+0000","last_fullsized":"2024-09-11T14:45:21.298538+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.358391+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:45:21.358391+0000","last_peered":"2024-09-11T14:45:21.358391+0000","last_clean":"2024-09-11T14:45:21.358391+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:45:21.358391+0000","last_undegraded":"2024-09-11T14:45:21.358391+0000","last_fullsized":"2024-09-11T14:45:21.358391+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"31'1513","reported_seq":1561,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:09.305856+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:46:09.305856+0000","last_peered":"2024-09-11T14:46:09.305856+0000","last_clean":"2024-09-11T14:46:09.305856+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:46:09.305856+0000","last_undegraded":"2024-09-11T14:46:09.305856+0000","last_fullsized":"2024-09-11T14:46:09.305856+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1513,"log_dups_size":0,"ondisk_log_size":1513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1513,"num_write_kb":51904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.302556+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:45:21.302556+0000","last_peered":"2024-09-11T14:45:21.302556+0000","last_clean":"2024-09-11T14:45:21.302556+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:45:21.302556+0000","last_undegraded":"2024-09-11T14:45:21.302556+0000","last_fullsized":"2024-09-11T14:45:21.302556+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"31'1528","reported_seq":1576,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:09.322671+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:46:09.322671+0000","last_peered":"2024-09-11T14:46:09.322671+0000","last_clean":"2024-09-11T14:46:09.322671+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:46:09.322671+0000","last_undegraded":"2024-09-11T14:46:09.322671+0000","last_fullsized":"2024-09-11T14:46:09.322671+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1528,"log_dups_size":0,"ondisk_log_size":1528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6684672,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1528,"num_write_kb":52160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.354421+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:45:21.354421+0000","last_peered":"2024-09-11T14:45:21.354421+0000","last_clean":"2024-09-11T14:45:21.354421+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:45:21.354421+0000","last_undegraded":"2024-09-11T14:45:21.354421+0000","last_fullsized":"2024-09-11T14:45:21.354421+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"31'1527","reported_seq":1575,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:09.626282+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:46:09.626282+0000","last_peered":"2024-09-11T14:46:09.626282+0000","last_clean":"2024-09-11T14:46:09.626282+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:46:09.626282+0000","last_undegraded":"2024-09-11T14:46:09.626282+0000","last_fullsized":"2024-09-11T14:46:09.626282+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1527,"log_dups_size":0,"ondisk_log_size":1527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6488064,"num_objects":99,"num_object_clones":0,"num_object_copies":297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1527,"num_write_kb":52032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.303454+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:45:21.303454+0000","last_peered":"2024-09-11T14:45:21.303454+0000","last_clean":"2024-09-11T14:45:21.303454+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:45:21.303454+0000","last_undegraded":"2024-09-11T14:45:21.303454+0000","last_fullsized":"2024-09-11T14:45:21.303454+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"31'1504","reported_seq":1552,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:10.628407+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:46:10.628407+0000","last_peered":"2024-09-11T14:46:10.628407+0000","last_clean":"2024-09-11T14:46:10.628407+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:46:10.628407+0000","last_undegraded":"2024-09-11T14:46:10.628407+0000","last_fullsized":"2024-09-11T14:46:10.628407+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1504,"log_dups_size":0,"ondisk_log_size":1504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6160384,"num_objects":94,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1504,"num_write_kb":51136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"31'1559","reported_seq":1607,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:09.742429+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:46:09.742429+0000","last_peered":"2024-09-11T14:46:09.742429+0000","last_clean":"2024-09-11T14:46:09.742429+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:46:09.742429+0000","last_undegraded":"2024-09-11T14:46:09.742429+0000","last_fullsized":"2024-09-11T14:46:09.742429+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1559,"log_dups_size":0,"ondisk_log_size":1559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1559,"num_write_kb":53120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.308554+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:45:21.308554+0000","last_peered":"2024-09-11T14:45:21.308554+0000","last_clean":"2024-09-11T14:45:21.308554+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:45:21.308554+0000","last_undegraded":"2024-09-11T14:45:21.308554+0000","last_fullsized":"2024-09-11T14:45:21.308554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"31'1429","reported_seq":1477,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:10.872614+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:46:10.872614+0000","last_peered":"2024-09-11T14:46:10.872614+0000","last_clean":"2024-09-11T14:46:10.872614+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:46:10.872614+0000","last_undegraded":"2024-09-11T14:46:10.872614+0000","last_fullsized":"2024-09-11T14:46:10.872614+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1429,"log_dups_size":0,"ondisk_log_size":1429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4915200,"num_objects":75,"num_object_clones":0,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1429,"num_write_kb":48128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.299078+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:45:21.299078+0000","last_peered":"2024-09-11T14:45:21.299078+0000","last_clean":"2024-09-11T14:45:21.299078+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:45:21.299078+0000","last_undegraded":"2024-09-11T14:45:21.299078+0000","last_fullsized":"2024-09-11T14:45:21.299078+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"31'1549","reported_seq":1597,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:09.001138+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:46:09.001138+0000","last_peered":"2024-09-11T14:46:09.001138+0000","last_clean":"2024-09-11T14:46:09.001138+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:46:09.001138+0000","last_undegraded":"2024-09-11T14:46:09.001138+0000","last_fullsized":"2024-09-11T14:46:09.001138+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1549,"log_dups_size":0,"ondisk_log_size":1549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1549,"num_write_kb":52928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.292230+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:45:21.292230+0000","last_peered":"2024-09-11T14:45:21.292230+0000","last_clean":"2024-09-11T14:45:21.292230+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:45:21.292230+0000","last_undegraded":"2024-09-11T14:45:21.292230+0000","last_fullsized":"2024-09-11T14:45:21.292230+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"31'1546","reported_seq":1594,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:10.775077+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:46:10.775077+0000","last_peered":"2024-09-11T14:46:10.775077+0000","last_clean":"2024-09-11T14:46:10.775077+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:46:10.775077+0000","last_undegraded":"2024-09-11T14:46:10.775077+0000","last_fullsized":"2024-09-11T14:46:10.775077+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1546,"log_dups_size":0,"ondisk_log_size":1546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5373952,"num_objects":82,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1546,"num_write_kb":52096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.358988+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:45:21.358988+0000","last_peered":"2024-09-11T14:45:21.358988+0000","last_clean":"2024-09-11T14:45:21.358988+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:45:21.358988+0000","last_undegraded":"2024-09-11T14:45:21.358988+0000","last_fullsized":"2024-09-11T14:45:21.358988+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":100859928,"num_objects":1540,"num_object_clones":0,"num_object_copies":4620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":24668,"num_write_kb":838593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":213553070080,"internally_reserved":0,"allocated":658423808,"data_stored":658423808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24668,"ondisk_log_size":24668,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":213552939008,"internally_reserved":0,"allocated":658554880,"data_stored":658554880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53357404160,"internally_reserved":0,"allocated":195469312,"data_stored":195469312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673264,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":140892,"kb_used_data":140892,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52156836,"statfs":{"total":53552873472,"available":53408600064,"internally_reserved":0,"allocated":144273408,"data_stored":144273408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":195648,"kb_used_data":195648,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52102080,"statfs":{"total":53552873472,"available":53352529920,"internally_reserved":0,"allocated":200343552,"data_stored":200343552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":169964,"kb_used_data":169964,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52127764,"statfs":{"total":53552873472,"available":53378830336,"internally_reserved":0,"allocated":174043136,"data_stored":174043136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":154424,"kb_used_data":154424,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52143304,"statfs":{"total":53552873472,"available":53394743296,"internally_reserved":0,"allocated":158130176,"data_stored":158130176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53357404160,"internally_reserved":0,"allocated":195469312,"data_stored":195469312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53399232512,"internally_reserved":0,"allocated":153640960,"data_stored":153640960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53384527872,"internally_reserved":0,"allocated":168345600,"data_stored":168345600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53411643392,"internally_reserved":0,"allocated":141230080,"data_stored":141230080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53357535232,"internally_reserved":0,"allocated":195338240,"data_stored":195338240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":53399232512,"internally_reserved":0,"allocated":153640960,"data_stored":153640960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":53384527872,"internally_reserved":0,"allocated":168345600,"data_stored":168345600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":53411774464,"internally_reserved":0,"allocated":141099008,"data_stored":141099008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":53357535232,"internally_reserved":0,"allocated":195338240,"data_stored":195338240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:46:16.723 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:46:16.856 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:16.852+0000 7f7f5d24f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:16.862 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:16.860+0000 7f7f5d24f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:19.572 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:46:19.712 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:19.708+0000 7fde39991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:19.718 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:19.717+0000 7fde39991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:20.457 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:46:20.457 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:46:20.457 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:46:20.588 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:20.583+0000 7f445b751640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:20.593 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:20.592+0000 7f445b751640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:20.796 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:46:20.797 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:46:20.813 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":182,"stamp":"2024-09-11T14:46:20.776558+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":53019211,"num_objects":805,"num_object_clones":0,"num_object_copies":2415,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":25488,"num_write_kb":839701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25462,"ondisk_log_size":25462,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":373392,"kb_used_data":373392,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":208817520,"statfs":{"total":214211493888,"available":213829140480,"internally_reserved":0,"allocated":382353408,"data_stored":382353408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-73596928,"num_objects":-1123,"num_object_clones":0,"num_object_copies":-3369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1123,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002907"},"pg_stats":[{"pgid":"3.f","version":"31'1586","reported_seq":1634,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:19.770691+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:46:19.770691+0000","last_peered":"2024-09-11T14:46:19.770691+0000","last_clean":"2024-09-11T14:46:19.770691+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:46:19.770691+0000","last_undegraded":"2024-09-11T14:46:19.770691+0000","last_fullsized":"2024-09-11T14:46:19.770691+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1586,"log_dups_size":0,"ondisk_log_size":1586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2097152,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1586,"num_write_kb":51776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"31'1710","reported_seq":1758,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:15.229618+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:46:15.229618+0000","last_peered":"2024-09-11T14:46:15.229618+0000","last_clean":"2024-09-11T14:46:15.229618+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:46:15.229618+0000","last_undegraded":"2024-09-11T14:46:15.229618+0000","last_fullsized":"2024-09-11T14:46:15.229618+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1710,"log_dups_size":0,"ondisk_log_size":1710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4587520,"num_objects":70,"num_object_clones":0,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1710,"num_write_kb":56960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"31'1614","reported_seq":1663,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:18.490971+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:46:18.490971+0000","last_peered":"2024-09-11T14:46:18.490971+0000","last_clean":"2024-09-11T14:46:18.490971+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:46:18.490971+0000","last_undegraded":"2024-09-11T14:46:18.490971+0000","last_fullsized":"2024-09-11T14:46:18.490971+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1614,"log_dups_size":0,"ondisk_log_size":1614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3473432,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1614,"num_write_kb":53313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"31'1535","reported_seq":1583,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:19.365530+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:46:19.365530+0000","last_peered":"2024-09-11T14:46:19.365530+0000","last_clean":"2024-09-11T14:46:19.365530+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:46:19.365530+0000","last_undegraded":"2024-09-11T14:46:19.365530+0000","last_fullsized":"2024-09-11T14:46:19.365530+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1535,"log_dups_size":0,"ondisk_log_size":1535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2949120,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1535,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"31'1554","reported_seq":1602,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:19.563214+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:46:19.563214+0000","last_peered":"2024-09-11T14:46:19.563214+0000","last_clean":"2024-09-11T14:46:19.563214+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:46:19.563214+0000","last_undegraded":"2024-09-11T14:46:19.563214+0000","last_fullsized":"2024-09-11T14:46:19.563214+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1554,"log_dups_size":0,"ondisk_log_size":1554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3276800,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1554,"num_write_kb":51328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"31'1723","reported_seq":1771,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:19.662980+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:46:19.662980+0000","last_peered":"2024-09-11T14:46:19.662980+0000","last_clean":"2024-09-11T14:46:19.662980+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:46:19.662980+0000","last_undegraded":"2024-09-11T14:46:19.662980+0000","last_fullsized":"2024-09-11T14:46:19.662980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1723,"log_dups_size":0,"ondisk_log_size":1723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2949120,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1723,"num_write_kb":56576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"31'1630","reported_seq":1678,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:19.658629+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:46:19.658629+0000","last_peered":"2024-09-11T14:46:19.658629+0000","last_clean":"2024-09-11T14:46:19.658629+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:46:19.658629+0000","last_undegraded":"2024-09-11T14:46:19.658629+0000","last_fullsized":"2024-09-11T14:46:19.658629+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1630,"log_dups_size":0,"ondisk_log_size":1630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2883584,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1630,"num_write_kb":53568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"31'1555","reported_seq":1603,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:18.595405+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:46:18.595405+0000","last_peered":"2024-09-11T14:46:18.595405+0000","last_clean":"2024-09-11T14:46:18.595405+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:46:18.595405+0000","last_undegraded":"2024-09-11T14:46:18.595405+0000","last_fullsized":"2024-09-11T14:46:18.595405+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1555,"log_dups_size":0,"ondisk_log_size":1555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2555904,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1555,"num_write_kb":51008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":133,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.298538+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:45:21.298538+0000","last_peered":"2024-09-11T14:45:21.298538+0000","last_clean":"2024-09-11T14:45:21.298538+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:45:21.298538+0000","last_undegraded":"2024-09-11T14:45:21.298538+0000","last_fullsized":"2024-09-11T14:45:21.298538+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.358391+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:45:21.358391+0000","last_peered":"2024-09-11T14:45:21.358391+0000","last_clean":"2024-09-11T14:45:21.358391+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:45:21.358391+0000","last_undegraded":"2024-09-11T14:45:21.358391+0000","last_fullsized":"2024-09-11T14:45:21.358391+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"31'1569","reported_seq":1617,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:19.764030+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:46:19.764030+0000","last_peered":"2024-09-11T14:46:19.764030+0000","last_clean":"2024-09-11T14:46:19.764030+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:46:19.764030+0000","last_undegraded":"2024-09-11T14:46:19.764030+0000","last_fullsized":"2024-09-11T14:46:19.764030+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1569,"log_dups_size":0,"ondisk_log_size":1569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3473408,"num_objects":53,"num_object_clones":0,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1569,"num_write_kb":51904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.302556+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:45:21.302556+0000","last_peered":"2024-09-11T14:45:21.302556+0000","last_clean":"2024-09-11T14:45:21.302556+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:45:21.302556+0000","last_undegraded":"2024-09-11T14:45:21.302556+0000","last_fullsized":"2024-09-11T14:45:21.302556+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"31'1588","reported_seq":1636,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:19.132926+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:46:19.132926+0000","last_peered":"2024-09-11T14:46:19.132926+0000","last_clean":"2024-09-11T14:46:19.132926+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:46:19.132926+0000","last_undegraded":"2024-09-11T14:46:19.132926+0000","last_fullsized":"2024-09-11T14:46:19.132926+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1588,"log_dups_size":0,"ondisk_log_size":1588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2752512,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1588,"num_write_kb":52160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.354421+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:45:21.354421+0000","last_peered":"2024-09-11T14:45:21.354421+0000","last_clean":"2024-09-11T14:45:21.354421+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:45:21.354421+0000","last_undegraded":"2024-09-11T14:45:21.354421+0000","last_fullsized":"2024-09-11T14:45:21.354421+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"31'1571","reported_seq":1619,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:19.606139+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:46:19.606139+0000","last_peered":"2024-09-11T14:46:19.606139+0000","last_clean":"2024-09-11T14:46:19.606139+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:46:19.606139+0000","last_undegraded":"2024-09-11T14:46:19.606139+0000","last_fullsized":"2024-09-11T14:46:19.606139+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1571,"log_dups_size":0,"ondisk_log_size":1571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3604480,"num_objects":55,"num_object_clones":0,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1571,"num_write_kb":52032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.303454+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:45:21.303454+0000","last_peered":"2024-09-11T14:45:21.303454+0000","last_clean":"2024-09-11T14:45:21.303454+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:45:21.303454+0000","last_undegraded":"2024-09-11T14:45:21.303454+0000","last_fullsized":"2024-09-11T14:45:21.303454+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"31'1524","reported_seq":1572,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:15.503737+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:46:15.503737+0000","last_peered":"2024-09-11T14:46:15.503737+0000","last_clean":"2024-09-11T14:46:15.503737+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:46:15.503737+0000","last_undegraded":"2024-09-11T14:46:15.503737+0000","last_fullsized":"2024-09-11T14:46:15.503737+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1524,"log_dups_size":0,"ondisk_log_size":1524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4849664,"num_objects":74,"num_object_clones":0,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1524,"num_write_kb":51136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"31'1620","reported_seq":1668,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:19.456573+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:46:19.456573+0000","last_peered":"2024-09-11T14:46:19.456573+0000","last_clean":"2024-09-11T14:46:19.456573+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:46:19.456573+0000","last_undegraded":"2024-09-11T14:46:19.456573+0000","last_fullsized":"2024-09-11T14:46:19.456573+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1620,"log_dups_size":0,"ondisk_log_size":1620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2621440,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1620,"num_write_kb":53120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.308554+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:45:21.308554+0000","last_peered":"2024-09-11T14:45:21.308554+0000","last_clean":"2024-09-11T14:45:21.308554+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:45:21.308554+0000","last_undegraded":"2024-09-11T14:45:21.308554+0000","last_fullsized":"2024-09-11T14:45:21.308554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"31'1454","reported_seq":1502,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:15.406917+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:46:15.406917+0000","last_peered":"2024-09-11T14:46:15.406917+0000","last_clean":"2024-09-11T14:46:15.406917+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:46:15.406917+0000","last_undegraded":"2024-09-11T14:46:15.406917+0000","last_fullsized":"2024-09-11T14:46:15.406917+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1454,"log_dups_size":0,"ondisk_log_size":1454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3276800,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1454,"num_write_kb":48128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.299078+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:45:21.299078+0000","last_peered":"2024-09-11T14:45:21.299078+0000","last_clean":"2024-09-11T14:45:21.299078+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:45:21.299078+0000","last_undegraded":"2024-09-11T14:45:21.299078+0000","last_fullsized":"2024-09-11T14:45:21.299078+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"31'1606","reported_seq":1654,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:19.582481+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:46:19.582481+0000","last_peered":"2024-09-11T14:46:19.582481+0000","last_clean":"2024-09-11T14:46:19.582481+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:46:19.582481+0000","last_undegraded":"2024-09-11T14:46:19.582481+0000","last_fullsized":"2024-09-11T14:46:19.582481+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1606,"log_dups_size":0,"ondisk_log_size":1606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1606,"num_write_kb":52928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.292230+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:45:21.292230+0000","last_peered":"2024-09-11T14:45:21.292230+0000","last_clean":"2024-09-11T14:45:21.292230+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:45:21.292230+0000","last_undegraded":"2024-09-11T14:45:21.292230+0000","last_fullsized":"2024-09-11T14:45:21.292230+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"31'1568","reported_seq":1616,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:14.892424+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:46:14.892424+0000","last_peered":"2024-09-11T14:46:14.892424+0000","last_clean":"2024-09-11T14:46:14.892424+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:46:14.892424+0000","last_undegraded":"2024-09-11T14:46:14.892424+0000","last_fullsized":"2024-09-11T14:46:14.892424+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1568,"log_dups_size":0,"ondisk_log_size":1568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3932160,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1568,"num_write_kb":52096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.358988+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:45:21.358988+0000","last_peered":"2024-09-11T14:45:21.358988+0000","last_clean":"2024-09-11T14:45:21.358988+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:45:21.358988+0000","last_undegraded":"2024-09-11T14:45:21.358988+0000","last_fullsized":"2024-09-11T14:45:21.358988+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":52428824,"num_objects":801,"num_object_clones":0,"num_object_copies":2403,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":25407,"num_write_kb":838593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":213849743360,"internally_reserved":0,"allocated":361750528,"data_stored":361750528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25407,"ondisk_log_size":25407,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":213849219072,"internally_reserved":0,"allocated":362274816,"data_stored":362274816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53455556608,"internally_reserved":0,"allocated":97316864,"data_stored":97316864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673269,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":107088,"kb_used_data":107088,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52190640,"statfs":{"total":53552873472,"available":53443215360,"internally_reserved":0,"allocated":109658112,"data_stored":109658112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705978,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":100592,"kb_used_data":100592,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52197136,"statfs":{"total":53552873472,"available":53449867264,"internally_reserved":0,"allocated":103006208,"data_stored":103006208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705978,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":87108,"kb_used_data":87108,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52210620,"statfs":{"total":53552873472,"available":53463674880,"internally_reserved":0,"allocated":89198592,"data_stored":89198592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705978,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":78604,"kb_used_data":78604,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52219124,"statfs":{"total":53552873472,"available":53472382976,"internally_reserved":0,"allocated":80490496,"data_stored":80490496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53455556608,"internally_reserved":0,"allocated":97316864,"data_stored":97316864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53477253120,"internally_reserved":0,"allocated":75620352,"data_stored":75620352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53469605888,"internally_reserved":0,"allocated":83267584,"data_stored":83267584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53446803456,"internally_reserved":0,"allocated":106070016,"data_stored":106070016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53455556608,"internally_reserved":0,"allocated":97316864,"data_stored":97316864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":53477384192,"internally_reserved":0,"allocated":75489280,"data_stored":75489280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":53469736960,"internally_reserved":0,"allocated":83136512,"data_stored":83136512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":53447065600,"internally_reserved":0,"allocated":105807872,"data_stored":105807872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":53455556608,"internally_reserved":0,"allocated":97316864,"data_stored":97316864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:46:22.048 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:46:22.173 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:22.170+0000 7ff0bcc9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:22.179 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:22.178+0000 7ff0bcc9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:24.636 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:46:24.770 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:24.766+0000 7f9bd1d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:24.776 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:24.775+0000 7f9bd1d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:25.814 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:46:25.815 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:46:25.815 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:46:25.935 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:25.929+0000 7f422a391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:25.939 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:25.938+0000 7f422a391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:26.183 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:46:26.183 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:46:26.200 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":184,"stamp":"2024-09-11T14:46:24.777574+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":46465611,"num_objects":705,"num_object_clones":0,"num_object_copies":2115,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":25588,"num_write_kb":839701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25562,"ondisk_log_size":25562,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":337892,"kb_used_data":337892,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":208853020,"statfs":{"total":214211493888,"available":213865492480,"internally_reserved":0,"allocated":346001408,"data_stored":346001408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-54984704,"num_objects":-839,"num_object_clones":0,"num_object_copies":-2517,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":839,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002932"},"pg_stats":[{"pgid":"3.f","version":"31'1586","reported_seq":1634,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:19.770691+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:46:19.770691+0000","last_peered":"2024-09-11T14:46:19.770691+0000","last_clean":"2024-09-11T14:46:19.770691+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:46:19.770691+0000","last_undegraded":"2024-09-11T14:46:19.770691+0000","last_fullsized":"2024-09-11T14:46:19.770691+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1586,"log_dups_size":0,"ondisk_log_size":1586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2097152,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1586,"num_write_kb":51776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"31'1735","reported_seq":1783,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:20.683029+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:46:20.683029+0000","last_peered":"2024-09-11T14:46:20.683029+0000","last_clean":"2024-09-11T14:46:20.683029+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:46:20.683029+0000","last_undegraded":"2024-09-11T14:46:20.683029+0000","last_fullsized":"2024-09-11T14:46:20.683029+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1735,"log_dups_size":0,"ondisk_log_size":1735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2949120,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1735,"num_write_kb":56960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"31'1614","reported_seq":1663,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:18.490971+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:46:18.490971+0000","last_peered":"2024-09-11T14:46:18.490971+0000","last_clean":"2024-09-11T14:46:18.490971+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:46:18.490971+0000","last_undegraded":"2024-09-11T14:46:18.490971+0000","last_fullsized":"2024-09-11T14:46:18.490971+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1614,"log_dups_size":0,"ondisk_log_size":1614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3473432,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1614,"num_write_kb":53313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"31'1535","reported_seq":1583,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:19.365530+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:46:19.365530+0000","last_peered":"2024-09-11T14:46:19.365530+0000","last_clean":"2024-09-11T14:46:19.365530+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:46:19.365530+0000","last_undegraded":"2024-09-11T14:46:19.365530+0000","last_fullsized":"2024-09-11T14:46:19.365530+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1535,"log_dups_size":0,"ondisk_log_size":1535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2949120,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1535,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"31'1554","reported_seq":1602,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:19.563214+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:46:19.563214+0000","last_peered":"2024-09-11T14:46:19.563214+0000","last_clean":"2024-09-11T14:46:19.563214+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:46:19.563214+0000","last_undegraded":"2024-09-11T14:46:19.563214+0000","last_fullsized":"2024-09-11T14:46:19.563214+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1554,"log_dups_size":0,"ondisk_log_size":1554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3276800,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1554,"num_write_kb":51328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"31'1723","reported_seq":1771,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:19.662980+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:46:19.662980+0000","last_peered":"2024-09-11T14:46:19.662980+0000","last_clean":"2024-09-11T14:46:19.662980+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:46:19.662980+0000","last_undegraded":"2024-09-11T14:46:19.662980+0000","last_fullsized":"2024-09-11T14:46:19.662980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1723,"log_dups_size":0,"ondisk_log_size":1723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2949120,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1723,"num_write_kb":56576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"31'1630","reported_seq":1678,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:19.658629+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:46:19.658629+0000","last_peered":"2024-09-11T14:46:19.658629+0000","last_clean":"2024-09-11T14:46:19.658629+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:46:19.658629+0000","last_undegraded":"2024-09-11T14:46:19.658629+0000","last_fullsized":"2024-09-11T14:46:19.658629+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1630,"log_dups_size":0,"ondisk_log_size":1630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2883584,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1630,"num_write_kb":53568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"31'1555","reported_seq":1603,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:18.595405+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:46:18.595405+0000","last_peered":"2024-09-11T14:46:18.595405+0000","last_clean":"2024-09-11T14:46:18.595405+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:46:18.595405+0000","last_undegraded":"2024-09-11T14:46:18.595405+0000","last_fullsized":"2024-09-11T14:46:18.595405+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1555,"log_dups_size":0,"ondisk_log_size":1555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2555904,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1555,"num_write_kb":51008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":133,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.298538+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:45:21.298538+0000","last_peered":"2024-09-11T14:45:21.298538+0000","last_clean":"2024-09-11T14:45:21.298538+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:45:21.298538+0000","last_undegraded":"2024-09-11T14:45:21.298538+0000","last_fullsized":"2024-09-11T14:45:21.298538+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.358391+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:45:21.358391+0000","last_peered":"2024-09-11T14:45:21.358391+0000","last_clean":"2024-09-11T14:45:21.358391+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:45:21.358391+0000","last_undegraded":"2024-09-11T14:45:21.358391+0000","last_fullsized":"2024-09-11T14:45:21.358391+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"31'1569","reported_seq":1617,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:19.764030+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:46:19.764030+0000","last_peered":"2024-09-11T14:46:19.764030+0000","last_clean":"2024-09-11T14:46:19.764030+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:46:19.764030+0000","last_undegraded":"2024-09-11T14:46:19.764030+0000","last_fullsized":"2024-09-11T14:46:19.764030+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1569,"log_dups_size":0,"ondisk_log_size":1569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3473408,"num_objects":53,"num_object_clones":0,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1569,"num_write_kb":51904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.302556+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:45:21.302556+0000","last_peered":"2024-09-11T14:45:21.302556+0000","last_clean":"2024-09-11T14:45:21.302556+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:45:21.302556+0000","last_undegraded":"2024-09-11T14:45:21.302556+0000","last_fullsized":"2024-09-11T14:45:21.302556+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"31'1588","reported_seq":1636,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:19.132926+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:46:19.132926+0000","last_peered":"2024-09-11T14:46:19.132926+0000","last_clean":"2024-09-11T14:46:19.132926+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:46:19.132926+0000","last_undegraded":"2024-09-11T14:46:19.132926+0000","last_fullsized":"2024-09-11T14:46:19.132926+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1588,"log_dups_size":0,"ondisk_log_size":1588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2752512,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1588,"num_write_kb":52160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.354421+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:45:21.354421+0000","last_peered":"2024-09-11T14:45:21.354421+0000","last_clean":"2024-09-11T14:45:21.354421+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:45:21.354421+0000","last_undegraded":"2024-09-11T14:45:21.354421+0000","last_fullsized":"2024-09-11T14:45:21.354421+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"31'1571","reported_seq":1619,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:19.606139+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:46:19.606139+0000","last_peered":"2024-09-11T14:46:19.606139+0000","last_clean":"2024-09-11T14:46:19.606139+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:46:19.606139+0000","last_undegraded":"2024-09-11T14:46:19.606139+0000","last_fullsized":"2024-09-11T14:46:19.606139+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1571,"log_dups_size":0,"ondisk_log_size":1571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3604480,"num_objects":55,"num_object_clones":0,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1571,"num_write_kb":52032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.303454+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:45:21.303454+0000","last_peered":"2024-09-11T14:45:21.303454+0000","last_clean":"2024-09-11T14:45:21.303454+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:45:21.303454+0000","last_undegraded":"2024-09-11T14:45:21.303454+0000","last_fullsized":"2024-09-11T14:45:21.303454+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"31'1554","reported_seq":1602,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:20.849698+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:46:20.849698+0000","last_peered":"2024-09-11T14:46:20.849698+0000","last_clean":"2024-09-11T14:46:20.849698+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:46:20.849698+0000","last_undegraded":"2024-09-11T14:46:20.849698+0000","last_fullsized":"2024-09-11T14:46:20.849698+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1554,"log_dups_size":0,"ondisk_log_size":1554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2883584,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1554,"num_write_kb":51136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"31'1620","reported_seq":1668,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:19.456573+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:46:19.456573+0000","last_peered":"2024-09-11T14:46:19.456573+0000","last_clean":"2024-09-11T14:46:19.456573+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:46:19.456573+0000","last_undegraded":"2024-09-11T14:46:19.456573+0000","last_fullsized":"2024-09-11T14:46:19.456573+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1620,"log_dups_size":0,"ondisk_log_size":1620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2621440,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1620,"num_write_kb":53120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.308554+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:45:21.308554+0000","last_peered":"2024-09-11T14:45:21.308554+0000","last_clean":"2024-09-11T14:45:21.308554+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:45:21.308554+0000","last_undegraded":"2024-09-11T14:45:21.308554+0000","last_fullsized":"2024-09-11T14:45:21.308554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"31'1477","reported_seq":1525,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:20.184855+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:46:20.184855+0000","last_peered":"2024-09-11T14:46:20.184855+0000","last_clean":"2024-09-11T14:46:20.184855+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:46:20.184855+0000","last_undegraded":"2024-09-11T14:46:20.184855+0000","last_fullsized":"2024-09-11T14:46:20.184855+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1477,"log_dups_size":0,"ondisk_log_size":1477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1477,"num_write_kb":48128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.299078+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:45:21.299078+0000","last_peered":"2024-09-11T14:45:21.299078+0000","last_clean":"2024-09-11T14:45:21.299078+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:45:21.299078+0000","last_undegraded":"2024-09-11T14:45:21.299078+0000","last_fullsized":"2024-09-11T14:45:21.299078+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"31'1606","reported_seq":1654,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:19.582481+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:46:19.582481+0000","last_peered":"2024-09-11T14:46:19.582481+0000","last_clean":"2024-09-11T14:46:19.582481+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:46:19.582481+0000","last_undegraded":"2024-09-11T14:46:19.582481+0000","last_fullsized":"2024-09-11T14:46:19.582481+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1606,"log_dups_size":0,"ondisk_log_size":1606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1606,"num_write_kb":52928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.292230+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:45:21.292230+0000","last_peered":"2024-09-11T14:45:21.292230+0000","last_clean":"2024-09-11T14:45:21.292230+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:45:21.292230+0000","last_undegraded":"2024-09-11T14:45:21.292230+0000","last_fullsized":"2024-09-11T14:45:21.292230+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"31'1590","reported_seq":1638,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:20.199304+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:46:20.199304+0000","last_peered":"2024-09-11T14:46:20.199304+0000","last_clean":"2024-09-11T14:46:20.199304+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:46:20.199304+0000","last_undegraded":"2024-09-11T14:46:20.199304+0000","last_fullsized":"2024-09-11T14:46:20.199304+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1590,"log_dups_size":0,"ondisk_log_size":1590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1590,"num_write_kb":52096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.358988+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:45:21.358988+0000","last_peered":"2024-09-11T14:45:21.358988+0000","last_clean":"2024-09-11T14:45:21.358988+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:45:21.358988+0000","last_undegraded":"2024-09-11T14:45:21.358988+0000","last_fullsized":"2024-09-11T14:45:21.358988+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":45875224,"num_objects":701,"num_object_clones":0,"num_object_copies":2103,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":25507,"num_write_kb":838593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":213884620800,"internally_reserved":0,"allocated":326873088,"data_stored":326873088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25507,"ondisk_log_size":25507,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":213884358656,"internally_reserved":0,"allocated":327135232,"data_stored":327135232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53455556608,"internally_reserved":0,"allocated":97316864,"data_stored":97316864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673274,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":71588,"kb_used_data":71588,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52226140,"statfs":{"total":53552873472,"available":53479567360,"internally_reserved":0,"allocated":73306112,"data_stored":73306112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705978,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":100592,"kb_used_data":100592,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52197136,"statfs":{"total":53552873472,"available":53449867264,"internally_reserved":0,"allocated":103006208,"data_stored":103006208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705978,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":87108,"kb_used_data":87108,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52210620,"statfs":{"total":53552873472,"available":53463674880,"internally_reserved":0,"allocated":89198592,"data_stored":89198592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705978,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":78604,"kb_used_data":78604,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52219124,"statfs":{"total":53552873472,"available":53472382976,"internally_reserved":0,"allocated":80490496,"data_stored":80490496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53455556608,"internally_reserved":0,"allocated":97316864,"data_stored":97316864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53477253120,"internally_reserved":0,"allocated":75620352,"data_stored":75620352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53469605888,"internally_reserved":0,"allocated":83267584,"data_stored":83267584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53481943040,"internally_reserved":0,"allocated":70930432,"data_stored":70930432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53455556608,"internally_reserved":0,"allocated":97316864,"data_stored":97316864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":53477384192,"internally_reserved":0,"allocated":75489280,"data_stored":75489280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":53469736960,"internally_reserved":0,"allocated":83136512,"data_stored":83136512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":53481943040,"internally_reserved":0,"allocated":70930432,"data_stored":70930432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":53455556608,"internally_reserved":0,"allocated":97316864,"data_stored":97316864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:46:27.032 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:46:27.184 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:27.180+0000 7f5cc2cd1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:27.191 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:27.190+0000 7f5cc2cd1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:29.531 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:46:29.646 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:29.643+0000 7efe38a84640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:29.650 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:29.649+0000 7efe38a84640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:31.201 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-11T14:46:31.202 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-09-11T14:46:31.202 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:46:31.324 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:31.318+0000 7fd1a3d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:31.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:31.326+0000 7fd1a3d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:31.526 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:46:31.527 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:46:31.540 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":187,"stamp":"2024-09-11T14:46:30.778964+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":10551883,"num_objects":157,"num_object_clones":0,"num_object_copies":471,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":47,"num_write":26136,"num_write_kb":839701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26110,"ondisk_log_size":26110,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":118904,"kb_used_data":118904,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209072008,"statfs":{"total":214211493888,"available":214089736192,"internally_reserved":0,"allocated":121757696,"data_stored":121757696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-63569920,"num_objects":-970,"num_object_clones":0,"num_object_copies":-2910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":970,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002835"},"pg_stats":[{"pgid":"3.f","version":"31'1614","reported_seq":1662,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:29.118893+0000","last_change":"2024-09-11T14:41:20.084656+0000","last_active":"2024-09-11T14:46:29.118893+0000","last_peered":"2024-09-11T14:46:29.118893+0000","last_clean":"2024-09-11T14:46:29.118893+0000","last_became_active":"2024-09-11T14:41:20.074370+0000","last_became_peered":"2024-09-11T14:41:20.074370+0000","last_unstale":"2024-09-11T14:46:29.118893+0000","last_undegraded":"2024-09-11T14:46:29.118893+0000","last_fullsized":"2024-09-11T14:46:29.118893+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1614,"log_dups_size":0,"ondisk_log_size":1614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262144,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1614,"num_write_kb":51776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"31'1756","reported_seq":1804,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:24.832776+0000","last_change":"2024-09-11T14:41:20.005266+0000","last_active":"2024-09-11T14:46:24.832776+0000","last_peered":"2024-09-11T14:46:24.832776+0000","last_clean":"2024-09-11T14:46:24.832776+0000","last_became_active":"2024-09-11T14:41:19.993612+0000","last_became_peered":"2024-09-11T14:41:19.993612+0000","last_unstale":"2024-09-11T14:46:24.832776+0000","last_undegraded":"2024-09-11T14:46:24.832776+0000","last_fullsized":"2024-09-11T14:46:24.832776+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1756,"log_dups_size":0,"ondisk_log_size":1756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1756,"num_write_kb":56960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"31'1659","reported_seq":1708,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:29.320268+0000","last_change":"2024-09-11T14:41:20.031723+0000","last_active":"2024-09-11T14:46:29.320268+0000","last_peered":"2024-09-11T14:46:29.320268+0000","last_clean":"2024-09-11T14:46:29.320268+0000","last_became_active":"2024-09-11T14:41:20.005783+0000","last_became_peered":"2024-09-11T14:41:20.005783+0000","last_unstale":"2024-09-11T14:46:29.320268+0000","last_undegraded":"2024-09-11T14:46:29.320268+0000","last_fullsized":"2024-09-11T14:46:29.320268+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1659,"log_dups_size":0,"ondisk_log_size":1659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":524312,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1659,"num_write_kb":53313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"31'1572","reported_seq":1620,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:29.012876+0000","last_change":"2024-09-11T14:41:20.097928+0000","last_active":"2024-09-11T14:46:29.012876+0000","last_peered":"2024-09-11T14:46:29.012876+0000","last_clean":"2024-09-11T14:46:29.012876+0000","last_became_active":"2024-09-11T14:41:20.081700+0000","last_became_peered":"2024-09-11T14:41:20.081700+0000","last_unstale":"2024-09-11T14:46:29.012876+0000","last_undegraded":"2024-09-11T14:46:29.012876+0000","last_fullsized":"2024-09-11T14:46:29.012876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1572,"log_dups_size":0,"ondisk_log_size":1572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":524288,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1572,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"31'1596","reported_seq":1644,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:29.016328+0000","last_change":"2024-09-11T14:41:20.035402+0000","last_active":"2024-09-11T14:46:29.016328+0000","last_peered":"2024-09-11T14:46:29.016328+0000","last_clean":"2024-09-11T14:46:29.016328+0000","last_became_active":"2024-09-11T14:41:19.977897+0000","last_became_peered":"2024-09-11T14:41:19.977897+0000","last_unstale":"2024-09-11T14:46:29.016328+0000","last_undegraded":"2024-09-11T14:46:29.016328+0000","last_fullsized":"2024-09-11T14:46:29.016328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1596,"log_dups_size":0,"ondisk_log_size":1596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":524288,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1596,"num_write_kb":51328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"31'1764","reported_seq":1812,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:29.771952+0000","last_change":"2024-09-11T14:41:20.008116+0000","last_active":"2024-09-11T14:46:29.771952+0000","last_peered":"2024-09-11T14:46:29.771952+0000","last_clean":"2024-09-11T14:46:29.771952+0000","last_became_active":"2024-09-11T14:41:19.986200+0000","last_became_peered":"2024-09-11T14:41:19.986200+0000","last_unstale":"2024-09-11T14:46:29.771952+0000","last_undegraded":"2024-09-11T14:46:29.771952+0000","last_fullsized":"2024-09-11T14:46:29.771952+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1764,"log_dups_size":0,"ondisk_log_size":1764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262144,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1764,"num_write_kb":56576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"31'1667","reported_seq":1715,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:28.995132+0000","last_change":"2024-09-11T14:41:20.022923+0000","last_active":"2024-09-11T14:46:28.995132+0000","last_peered":"2024-09-11T14:46:28.995132+0000","last_clean":"2024-09-11T14:46:28.995132+0000","last_became_active":"2024-09-11T14:41:20.014305+0000","last_became_peered":"2024-09-11T14:41:20.014305+0000","last_unstale":"2024-09-11T14:46:28.995132+0000","last_undegraded":"2024-09-11T14:46:28.995132+0000","last_fullsized":"2024-09-11T14:46:28.995132+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1667,"log_dups_size":0,"ondisk_log_size":1667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458752,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1667,"num_write_kb":53568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"31'1590","reported_seq":1638,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:29.395178+0000","last_change":"2024-09-11T14:41:19.960606+0000","last_active":"2024-09-11T14:46:29.395178+0000","last_peered":"2024-09-11T14:46:29.395178+0000","last_clean":"2024-09-11T14:46:29.395178+0000","last_became_active":"2024-09-11T14:41:19.947839+0000","last_became_peered":"2024-09-11T14:41:19.947839+0000","last_unstale":"2024-09-11T14:46:29.395178+0000","last_undegraded":"2024-09-11T14:46:29.395178+0000","last_fullsized":"2024-09-11T14:46:29.395178+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1590,"log_dups_size":0,"ondisk_log_size":1590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262144,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1590,"num_write_kb":51008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":133,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.298538+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:45:21.298538+0000","last_peered":"2024-09-11T14:45:21.298538+0000","last_clean":"2024-09-11T14:45:21.298538+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:45:21.298538+0000","last_undegraded":"2024-09-11T14:45:21.298538+0000","last_fullsized":"2024-09-11T14:45:21.298538+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.358391+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:45:21.358391+0000","last_peered":"2024-09-11T14:45:21.358391+0000","last_clean":"2024-09-11T14:45:21.358391+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:45:21.358391+0000","last_undegraded":"2024-09-11T14:45:21.358391+0000","last_fullsized":"2024-09-11T14:45:21.358391+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"31'1612","reported_seq":1660,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:29.814783+0000","last_change":"2024-09-11T14:41:20.014218+0000","last_active":"2024-09-11T14:46:29.814783+0000","last_peered":"2024-09-11T14:46:29.814783+0000","last_clean":"2024-09-11T14:46:29.814783+0000","last_became_active":"2024-09-11T14:41:20.006168+0000","last_became_peered":"2024-09-11T14:41:20.006168+0000","last_unstale":"2024-09-11T14:46:29.814783+0000","last_undegraded":"2024-09-11T14:46:29.814783+0000","last_fullsized":"2024-09-11T14:46:29.814783+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1612,"log_dups_size":0,"ondisk_log_size":1612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":655360,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1612,"num_write_kb":51904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.302556+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:45:21.302556+0000","last_peered":"2024-09-11T14:45:21.302556+0000","last_clean":"2024-09-11T14:45:21.302556+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:45:21.302556+0000","last_undegraded":"2024-09-11T14:45:21.302556+0000","last_fullsized":"2024-09-11T14:45:21.302556+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"31'1624","reported_seq":1672,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:29.346329+0000","last_change":"2024-09-11T14:41:20.023893+0000","last_active":"2024-09-11T14:46:29.346329+0000","last_peered":"2024-09-11T14:46:29.346329+0000","last_clean":"2024-09-11T14:46:29.346329+0000","last_became_active":"2024-09-11T14:41:20.010462+0000","last_became_peered":"2024-09-11T14:41:20.010462+0000","last_unstale":"2024-09-11T14:46:29.346329+0000","last_undegraded":"2024-09-11T14:46:29.346329+0000","last_fullsized":"2024-09-11T14:46:29.346329+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1624,"log_dups_size":0,"ondisk_log_size":1624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1624,"num_write_kb":52160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.354421+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:45:21.354421+0000","last_peered":"2024-09-11T14:45:21.354421+0000","last_clean":"2024-09-11T14:45:21.354421+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:45:21.354421+0000","last_undegraded":"2024-09-11T14:45:21.354421+0000","last_fullsized":"2024-09-11T14:45:21.354421+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"31'1621","reported_seq":1669,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:29.116653+0000","last_change":"2024-09-11T14:41:20.125542+0000","last_active":"2024-09-11T14:46:29.116653+0000","last_peered":"2024-09-11T14:46:29.116653+0000","last_clean":"2024-09-11T14:46:29.116653+0000","last_became_active":"2024-09-11T14:41:20.117856+0000","last_became_peered":"2024-09-11T14:41:20.117856+0000","last_unstale":"2024-09-11T14:46:29.116653+0000","last_undegraded":"2024-09-11T14:46:29.116653+0000","last_fullsized":"2024-09-11T14:46:29.116653+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1621,"log_dups_size":0,"ondisk_log_size":1621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1621,"num_write_kb":52032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.303454+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:45:21.303454+0000","last_peered":"2024-09-11T14:45:21.303454+0000","last_clean":"2024-09-11T14:45:21.303454+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:45:21.303454+0000","last_undegraded":"2024-09-11T14:45:21.303454+0000","last_fullsized":"2024-09-11T14:45:21.303454+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"31'1578","reported_seq":1626,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:25.499991+0000","last_change":"2024-09-11T14:41:19.992269+0000","last_active":"2024-09-11T14:46:25.499991+0000","last_peered":"2024-09-11T14:46:25.499991+0000","last_clean":"2024-09-11T14:46:25.499991+0000","last_became_active":"2024-09-11T14:41:19.949307+0000","last_became_peered":"2024-09-11T14:41:19.949307+0000","last_unstale":"2024-09-11T14:46:25.499991+0000","last_undegraded":"2024-09-11T14:46:25.499991+0000","last_fullsized":"2024-09-11T14:46:25.499991+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1578,"num_write_kb":51136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"31'1655","reported_seq":1703,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:29.043705+0000","last_change":"2024-09-11T14:41:20.149946+0000","last_active":"2024-09-11T14:46:29.043705+0000","last_peered":"2024-09-11T14:46:29.043705+0000","last_clean":"2024-09-11T14:46:29.043705+0000","last_became_active":"2024-09-11T14:41:20.138803+0000","last_became_peered":"2024-09-11T14:41:20.138803+0000","last_unstale":"2024-09-11T14:46:29.043705+0000","last_undegraded":"2024-09-11T14:46:29.043705+0000","last_fullsized":"2024-09-11T14:46:29.043705+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1655,"log_dups_size":0,"ondisk_log_size":1655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1655,"num_write_kb":53120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.308554+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:45:21.308554+0000","last_peered":"2024-09-11T14:45:21.308554+0000","last_clean":"2024-09-11T14:45:21.308554+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:45:21.308554+0000","last_undegraded":"2024-09-11T14:45:21.308554+0000","last_fullsized":"2024-09-11T14:45:21.308554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"31'1491","reported_seq":1539,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:25.861095+0000","last_change":"2024-09-11T14:41:20.046526+0000","last_active":"2024-09-11T14:46:25.861095+0000","last_peered":"2024-09-11T14:46:25.861095+0000","last_clean":"2024-09-11T14:46:25.861095+0000","last_became_active":"2024-09-11T14:41:20.033623+0000","last_became_peered":"2024-09-11T14:41:20.033623+0000","last_unstale":"2024-09-11T14:46:25.861095+0000","last_undegraded":"2024-09-11T14:46:25.861095+0000","last_fullsized":"2024-09-11T14:46:25.861095+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1491,"log_dups_size":0,"ondisk_log_size":1491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":851968,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1491,"num_write_kb":48128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.299078+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:45:21.299078+0000","last_peered":"2024-09-11T14:45:21.299078+0000","last_clean":"2024-09-11T14:45:21.299078+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:45:21.299078+0000","last_undegraded":"2024-09-11T14:45:21.299078+0000","last_fullsized":"2024-09-11T14:45:21.299078+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"31'1645","reported_seq":1693,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:29.583714+0000","last_change":"2024-09-11T14:41:20.043546+0000","last_active":"2024-09-11T14:46:29.583714+0000","last_peered":"2024-09-11T14:46:29.583714+0000","last_clean":"2024-09-11T14:46:29.583714+0000","last_became_active":"2024-09-11T14:41:19.993565+0000","last_became_peered":"2024-09-11T14:41:19.993565+0000","last_unstale":"2024-09-11T14:46:29.583714+0000","last_undegraded":"2024-09-11T14:46:29.583714+0000","last_fullsized":"2024-09-11T14:46:29.583714+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1645,"log_dups_size":0,"ondisk_log_size":1645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":589824,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1645,"num_write_kb":52928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.292230+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:45:21.292230+0000","last_peered":"2024-09-11T14:45:21.292230+0000","last_clean":"2024-09-11T14:45:21.292230+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:45:21.292230+0000","last_undegraded":"2024-09-11T14:45:21.292230+0000","last_fullsized":"2024-09-11T14:45:21.292230+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"31'1611","reported_seq":1659,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:46:25.661198+0000","last_change":"2024-09-11T14:41:20.087011+0000","last_active":"2024-09-11T14:46:25.661198+0000","last_peered":"2024-09-11T14:46:25.661198+0000","last_clean":"2024-09-11T14:46:25.661198+0000","last_became_active":"2024-09-11T14:41:20.075527+0000","last_became_peered":"2024-09-11T14:41:20.075527+0000","last_unstale":"2024-09-11T14:46:25.661198+0000","last_undegraded":"2024-09-11T14:46:25.661198+0000","last_fullsized":"2024-09-11T14:46:25.661198+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:18.807024+0000","last_clean_scrub_stamp":"2024-09-11T14:41:18.807024+0000","objects_scrubbed":0,"log_size":1611,"log_dups_size":0,"ondisk_log_size":1611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1611,"num_write_kb":52096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T14:45:21.358988+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:45:21.358988+0000","last_peered":"2024-09-11T14:45:21.358988+0000","last_clean":"2024-09-11T14:45:21.358988+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:45:21.358988+0000","last_undegraded":"2024-09-11T14:45:21.358988+0000","last_fullsized":"2024-09-11T14:45:21.358988+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":9961496,"num_objects":153,"num_object_clones":0,"num_object_copies":459,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":26055,"num_write_kb":838593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214100230144,"internally_reserved":0,"allocated":111263744,"data_stored":111263744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26055,"ondisk_log_size":26055,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214100230144,"internally_reserved":0,"allocated":111263744,"data_stored":111263744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53525512192,"internally_reserved":0,"allocated":27361280,"data_stored":27361280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673279,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":42288,"kb_used_data":42288,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52255440,"statfs":{"total":53552873472,"available":53509570560,"internally_reserved":0,"allocated":43302912,"data_stored":43302912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705988,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":29956,"kb_used_data":29956,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52267772,"statfs":{"total":53552873472,"available":53522198528,"internally_reserved":0,"allocated":30674944,"data_stored":30674944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705988,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":24060,"kb_used_data":24060,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52273668,"statfs":{"total":53552873472,"available":53528236032,"internally_reserved":0,"allocated":24637440,"data_stored":24637440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705988,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":22600,"kb_used_data":22600,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52275128,"statfs":{"total":53552873472,"available":53529731072,"internally_reserved":0,"allocated":23142400,"data_stored":23142400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53525512192,"internally_reserved":0,"allocated":27361280,"data_stored":27361280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53531729920,"internally_reserved":0,"allocated":21143552,"data_stored":21143552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53531185152,"internally_reserved":0,"allocated":21688320,"data_stored":21688320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53511802880,"internally_reserved":0,"allocated":41070592,"data_stored":41070592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53525512192,"internally_reserved":0,"allocated":27361280,"data_stored":27361280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53552873472,"available":53531729920,"internally_reserved":0,"allocated":21143552,"data_stored":21143552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53552873472,"available":53531185152,"internally_reserved":0,"allocated":21688320,"data_stored":21688320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53552873472,"available":53511802880,"internally_reserved":0,"allocated":41070592,"data_stored":41070592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53552873472,"available":53525512192,"internally_reserved":0,"allocated":27361280,"data_stored":27361280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:46:31.996 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T14:46:32.127 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:32.124+0000 7fdacee74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:32.131 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:32.130+0000 7fdacee74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:33.419 INFO:tasks.radosbench.radosbench.0.smithi181.stdout:Removed 13103 objects 2024-09-11T14:46:33.420 INFO:tasks.radosbench.radosbench.0.smithi181.stdout:Clean up completed and total clean up time :161.461 2024-09-11T14:46:33.749 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-11T14:46:33.750 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-11T14:46:33.864 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:33.854+0000 7fa1768d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:33.868 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:33.866+0000 7fa1768d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:34.357 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T14:46:34.358 INFO:teuthology.orchestra.run.smithi181.stderr:pool 'unique_pool_0' removed 2024-09-11T14:46:34.375 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-11T14:46:34.384 INFO:tasks.thrashosds:joining thrashosds 2024-09-11T14:46:34.521 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:34.517+0000 7f25ac151640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:34.525 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:34.524+0000 7f25ac151640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:36.543 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-11T14:46:36.663 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:36.660+0000 7fc881711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:36.667 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:36.665+0000 7fc881711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:36.681 INFO:teuthology.orchestra.run.smithi181.stdout:{} 2024-09-11T14:46:36.682 INFO:teuthology.orchestra.run.smithi181.stderr:mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-11T14:46:36.693 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-11T14:46:36.803 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:36.795+0000 7fd04e391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:36.806 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:36.804+0000 7fd04e391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:37.370 INFO:teuthology.orchestra.run.smithi181.stderr:reweighted osd.0 to 1 (10000) 2024-09-11T14:46:37.387 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-11T14:46:37.500 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:37.492+0000 7f8844d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:37.503 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:37.502+0000 7f8844d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:38.375 INFO:teuthology.orchestra.run.smithi181.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-11T14:46:38.392 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-11T14:46:38.506 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:38.498+0000 7f8f5d2a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:38.510 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:38.508+0000 7f8f5d2a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:39.382 INFO:teuthology.orchestra.run.smithi181.stderr:reweighted osd.1 to 1 (10000) 2024-09-11T14:46:39.406 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-11T14:46:39.520 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:39.512+0000 7fadca863640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:39.523 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:39.522+0000 7fadca863640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:40.383 INFO:teuthology.orchestra.run.smithi181.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-11T14:46:40.401 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-11T14:46:40.514 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:40.506+0000 7f80bad91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:40.517 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:40.516+0000 7f80bad91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:41.390 INFO:teuthology.orchestra.run.smithi181.stderr:reweighted osd.2 to 1 (10000) 2024-09-11T14:46:41.415 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-11T14:46:41.527 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:41.520+0000 7fb709c4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:41.531 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:41.530+0000 7fb709c4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:42.391 INFO:teuthology.orchestra.run.smithi181.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-11T14:46:42.409 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-11T14:46:42.524 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:42.516+0000 7eff64991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:42.527 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:42.526+0000 7eff64991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:43.397 INFO:teuthology.orchestra.run.smithi181.stderr:reweighted osd.3 to 1 (10000) 2024-09-11T14:46:43.419 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-11T14:46:43.533 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:43.526+0000 7fa428ed1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:43.537 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:43.536+0000 7fa428ed1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:44.475 INFO:teuthology.orchestra.run.smithi181.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-11T14:46:44.498 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-11T14:46:44.498 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-11T14:46:44.498 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T14:46:44.609 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:44.603+0000 7f6c0c391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:44.612 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:44.611+0000 7f6c0c391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:44.800 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:46:44.800 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":40,"fsid":"0f7e90de-623e-43a2-a836-b493e06f03fc","created":"2024-09-11T14:40:58.371084+0000","modified":"2024-09-11T14:46:44.471423+0000","last_up_change":"2024-09-11T14:41:05.766940+0000","last_in_change":"2024-09-11T14:41:00.494637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T14:41:06.733924+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T14:41:08.627066+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"05592c04-44ae-46d8-bcf7-a865d8397d5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":920133847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6803","nonce":920133847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6809","nonce":920133847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6807","nonce":920133847}]},"public_addr":"172.21.15.181:6804/920133847","cluster_addr":"172.21.15.181:6803/920133847","heartbeat_back_addr":"172.21.15.181:6809/920133847","heartbeat_front_addr":"172.21.15.181:6807/920133847","state":["exists","up"]},{"osd":1,"uuid":"70bdeffe-ad6e-4417-92f7-6021f541c690","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6801","nonce":3969919471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3969919471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3969919471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6819","nonce":3969919471}]},"public_addr":"172.21.15.181:6801/3969919471","cluster_addr":"172.21.15.181:6802/3969919471","heartbeat_back_addr":"172.21.15.181:6820/3969919471","heartbeat_front_addr":"172.21.15.181:6819/3969919471","state":["exists","up"]},{"osd":2,"uuid":"d3fd0624-0127-4f26-ba89-ca34a2c562ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6811","nonce":692234911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":692234911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":692234911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":692234911}]},"public_addr":"172.21.15.203:6811/692234911","cluster_addr":"172.21.15.203:6810/692234911","heartbeat_back_addr":"172.21.15.203:6808/692234911","heartbeat_front_addr":"172.21.15.203:6809/692234911","state":["exists","up"]},{"osd":3,"uuid":"24bbbdd2-a931-4cc2-a24d-61d131ece2c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6803","nonce":667507199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":667507199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":667507199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6805","nonce":667507199}]},"public_addr":"172.21.15.203:6803/667507199","cluster_addr":"172.21.15.203:6802/667507199","heartbeat_back_addr":"172.21.15.203:6804/667507199","heartbeat_front_addr":"172.21.15.203:6805/667507199","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T14:46:44.813 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-11T14:46:44.814 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T14:46:44.923 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:44.918+0000 7f4639f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:44.927 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:44.926+0000 7f4639f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:45.111 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:46:45.111 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":40,"fsid":"0f7e90de-623e-43a2-a836-b493e06f03fc","created":"2024-09-11T14:40:58.371084+0000","modified":"2024-09-11T14:46:44.471423+0000","last_up_change":"2024-09-11T14:41:05.766940+0000","last_in_change":"2024-09-11T14:41:00.494637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T14:41:06.733924+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T14:41:08.627066+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"05592c04-44ae-46d8-bcf7-a865d8397d5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":920133847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6803","nonce":920133847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6809","nonce":920133847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6807","nonce":920133847}]},"public_addr":"172.21.15.181:6804/920133847","cluster_addr":"172.21.15.181:6803/920133847","heartbeat_back_addr":"172.21.15.181:6809/920133847","heartbeat_front_addr":"172.21.15.181:6807/920133847","state":["exists","up"]},{"osd":1,"uuid":"70bdeffe-ad6e-4417-92f7-6021f541c690","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6801","nonce":3969919471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3969919471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3969919471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6819","nonce":3969919471}]},"public_addr":"172.21.15.181:6801/3969919471","cluster_addr":"172.21.15.181:6802/3969919471","heartbeat_back_addr":"172.21.15.181:6820/3969919471","heartbeat_front_addr":"172.21.15.181:6819/3969919471","state":["exists","up"]},{"osd":2,"uuid":"d3fd0624-0127-4f26-ba89-ca34a2c562ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6811","nonce":692234911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":692234911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":692234911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":692234911}]},"public_addr":"172.21.15.203:6811/692234911","cluster_addr":"172.21.15.203:6810/692234911","heartbeat_back_addr":"172.21.15.203:6808/692234911","heartbeat_front_addr":"172.21.15.203:6809/692234911","state":["exists","up"]},{"osd":3,"uuid":"24bbbdd2-a931-4cc2-a24d-61d131ece2c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6803","nonce":667507199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":667507199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":667507199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6805","nonce":667507199}]},"public_addr":"172.21.15.203:6803/667507199","cluster_addr":"172.21.15.203:6802/667507199","heartbeat_back_addr":"172.21.15.203:6804/667507199","heartbeat_front_addr":"172.21.15.203:6805/667507199","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T14:46:45.125 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-11T14:46:45.125 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-11T14:46:45.126 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-11T14:46:45.126 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-11T14:46:45.248 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:45.241+0000 7f1e42f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:45.249 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:45.242+0000 7fef07f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:45.249 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:45.242+0000 7fa0ee65c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:45.249 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:45.242+0000 7fb371f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:45.253 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:45.252+0000 7f1e42f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:45.253 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:45.252+0000 7fa0ee65c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:45.254 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:45.252+0000 7fef07f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:45.254 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:45.253+0000 7fb371f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:45.289 INFO:teuthology.orchestra.run.smithi181.stdout:38654706003 2024-09-11T14:46:45.289 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T14:46:45.290 INFO:teuthology.orchestra.run.smithi181.stdout:38654706004 2024-09-11T14:46:45.291 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T14:46:45.291 INFO:teuthology.orchestra.run.smithi181.stdout:42949673298 2024-09-11T14:46:45.292 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T14:46:45.293 INFO:teuthology.orchestra.run.smithi181.stdout:38654706004 2024-09-11T14:46:45.293 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T14:46:45.408 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:45.401+0000 7f8c83f51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:45.408 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:45.401+0000 7f65d3c63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:45.408 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:45.401+0000 7f7e15711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:45.413 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:45.411+0000 7f65d3c63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:45.413 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:45.412+0000 7f8c83f51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:45.413 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:45.412+0000 7f7e15711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:45.464 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:45.458+0000 7fd47cf51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:45.469 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:45.468+0000 7fd47cf51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:45.617 INFO:teuthology.orchestra.run.smithi181.stdout:38654705998 2024-09-11T14:46:45.626 INFO:teuthology.orchestra.run.smithi181.stdout:38654705998 2024-09-11T14:46:45.637 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654706004 got 38654705998 for osd.1 2024-09-11T14:46:45.639 INFO:teuthology.orchestra.run.smithi181.stdout:42949673294 2024-09-11T14:46:45.641 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654706003 got 38654705998 for osd.0 2024-09-11T14:46:45.653 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949673298 got 42949673294 for osd.2 2024-09-11T14:46:45.674 INFO:teuthology.orchestra.run.smithi181.stdout:38654705998 2024-09-11T14:46:45.688 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654706004 got 38654705998 for osd.3 2024-09-11T14:46:46.638 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T14:46:46.641 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T14:46:46.654 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T14:46:46.690 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T14:46:46.758 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:46.750+0000 7f491ad91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:46.764 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:46.763+0000 7f491ad91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:46.800 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:46.793+0000 7fb46b511640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:46.803 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:46.796+0000 7fb9d546c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:46.804 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:46.803+0000 7fb46b511640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:46.808 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:46.807+0000 7fb9d546c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:46.847 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:46.840+0000 7ff167511640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:46.851 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:46.850+0000 7ff167511640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:46.968 INFO:teuthology.orchestra.run.smithi181.stdout:38654705998 2024-09-11T14:46:46.982 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654706004 got 38654705998 for osd.1 2024-09-11T14:46:47.016 INFO:teuthology.orchestra.run.smithi181.stdout:42949673294 2024-09-11T14:46:47.030 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949673298 got 42949673294 for osd.2 2024-09-11T14:46:47.034 INFO:teuthology.orchestra.run.smithi181.stdout:38654705998 2024-09-11T14:46:47.049 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654706003 got 38654705998 for osd.0 2024-09-11T14:46:47.067 INFO:teuthology.orchestra.run.smithi181.stdout:38654705998 2024-09-11T14:46:47.081 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654706004 got 38654705998 for osd.3 2024-09-11T14:46:47.984 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T14:46:48.031 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T14:46:48.049 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T14:46:48.081 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T14:46:48.111 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:48.103+0000 7f19cd511640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:48.116 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:48.115+0000 7f19cd511640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:48.154 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:48.147+0000 7fb4b1ad1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:48.159 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:48.158+0000 7fb4b1ad1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:48.197 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:48.189+0000 7f615bb91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:48.201 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:48.200+0000 7f615bb91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:48.233 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:48.227+0000 7f8ff24bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:48.238 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:48.237+0000 7f8ff24bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:48.336 INFO:teuthology.orchestra.run.smithi181.stdout:38654706004 2024-09-11T14:46:48.354 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654706004 got 38654706004 for osd.1 2024-09-11T14:46:48.354 DEBUG:teuthology.parallel:result is None 2024-09-11T14:46:48.361 INFO:teuthology.orchestra.run.smithi181.stdout:42949673299 2024-09-11T14:46:48.376 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949673298 got 42949673299 for osd.2 2024-09-11T14:46:48.376 DEBUG:teuthology.parallel:result is None 2024-09-11T14:46:48.409 INFO:teuthology.orchestra.run.smithi181.stdout:38654706003 2024-09-11T14:46:48.424 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654706003 got 38654706003 for osd.0 2024-09-11T14:46:48.424 DEBUG:teuthology.parallel:result is None 2024-09-11T14:46:48.456 INFO:teuthology.orchestra.run.smithi181.stdout:38654706004 2024-09-11T14:46:48.470 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654706004 got 38654706004 for osd.3 2024-09-11T14:46:48.470 DEBUG:teuthology.parallel:result is None 2024-09-11T14:46:48.470 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-11T14:46:48.470 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:46:48.618 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:48.613+0000 7ff136d11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:48.622 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:48.621+0000 7ff136d11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:48.806 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:46:48.807 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:46:48.820 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-09-11T14:46:48.783293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":8528,"kb_used_data":8528,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209182384,"statfs":{"total":214211493888,"available":214202761216,"internally_reserved":0,"allocated":8732672,"data_stored":8732672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.391996"},"pg_stats":[{"pgid":"1.0","version":"13'52","reported_seq":160,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.508959+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:46:44.508959+0000","last_peered":"2024-09-11T14:46:44.508959+0000","last_clean":"2024-09-11T14:46:44.508959+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:46:44.508959+0000","last_undegraded":"2024-09-11T14:46:44.508959+0000","last_fullsized":"2024-09-11T14:46:44.508959+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":89,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.523820+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:46:44.523820+0000","last_peered":"2024-09-11T14:46:44.523820+0000","last_clean":"2024-09-11T14:46:44.523820+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:46:44.523820+0000","last_undegraded":"2024-09-11T14:46:44.523820+0000","last_fullsized":"2024-09-11T14:46:44.523820+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.514162+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:46:44.514162+0000","last_peered":"2024-09-11T14:46:44.514162+0000","last_clean":"2024-09-11T14:46:44.514162+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:46:44.514162+0000","last_undegraded":"2024-09-11T14:46:44.514162+0000","last_fullsized":"2024-09-11T14:46:44.514162+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.521574+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:46:44.521574+0000","last_peered":"2024-09-11T14:46:44.521574+0000","last_clean":"2024-09-11T14:46:44.521574+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:46:44.521574+0000","last_undegraded":"2024-09-11T14:46:44.521574+0000","last_fullsized":"2024-09-11T14:46:44.521574+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":90,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.514411+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:46:44.514411+0000","last_peered":"2024-09-11T14:46:44.514411+0000","last_clean":"2024-09-11T14:46:44.514411+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:46:44.514411+0000","last_undegraded":"2024-09-11T14:46:44.514411+0000","last_fullsized":"2024-09-11T14:46:44.514411+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.512373+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:46:44.512373+0000","last_peered":"2024-09-11T14:46:44.512373+0000","last_clean":"2024-09-11T14:46:44.512373+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:46:44.512373+0000","last_undegraded":"2024-09-11T14:46:44.512373+0000","last_fullsized":"2024-09-11T14:46:44.512373+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.510313+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:46:44.510313+0000","last_peered":"2024-09-11T14:46:44.510313+0000","last_clean":"2024-09-11T14:46:44.510313+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:46:44.510313+0000","last_undegraded":"2024-09-11T14:46:44.510313+0000","last_fullsized":"2024-09-11T14:46:44.510313+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.512782+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:46:44.512782+0000","last_peered":"2024-09-11T14:46:44.512782+0000","last_clean":"2024-09-11T14:46:44.512782+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:46:44.512782+0000","last_undegraded":"2024-09-11T14:46:44.512782+0000","last_fullsized":"2024-09-11T14:46:44.512782+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.523616+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:46:44.523616+0000","last_peered":"2024-09-11T14:46:44.523616+0000","last_clean":"2024-09-11T14:46:44.523616+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:46:44.523616+0000","last_undegraded":"2024-09-11T14:46:44.523616+0000","last_fullsized":"2024-09-11T14:46:44.523616+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214202736640,"internally_reserved":0,"allocated":8757248,"data_stored":8757248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53550514176,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673299,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2380,"kb_used_data":2380,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295348,"statfs":{"total":53552873472,"available":53550436352,"internally_reserved":0,"allocated":2437120,"data_stored":2437120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654706004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2304,"kb_used_data":2304,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295424,"statfs":{"total":53552873472,"available":53550514176,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1536,"kb_used_data":1536,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296192,"statfs":{"total":53552873472,"available":53551300608,"internally_reserved":0,"allocated":1572864,"data_stored":1572864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706003,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2308,"kb_used_data":2308,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295420,"statfs":{"total":53552873472,"available":53550510080,"internally_reserved":0,"allocated":2363392,"data_stored":2363392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53550514176,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53550485504,"internally_reserved":0,"allocated":2387968,"data_stored":2387968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551300608,"internally_reserved":0,"allocated":1572864,"data_stored":1572864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53550436352,"internally_reserved":0,"allocated":2437120,"data_stored":2437120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53550514176,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:46:48.821 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:46:48.929 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:48.924+0000 7f7314391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:48.933 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:48.931+0000 7f7314391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:49.116 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:46:49.116 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:46:49.129 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-09-11T14:46:48.783293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":8528,"kb_used_data":8528,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209182384,"statfs":{"total":214211493888,"available":214202761216,"internally_reserved":0,"allocated":8732672,"data_stored":8732672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.391996"},"pg_stats":[{"pgid":"1.0","version":"13'52","reported_seq":160,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.508959+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:46:44.508959+0000","last_peered":"2024-09-11T14:46:44.508959+0000","last_clean":"2024-09-11T14:46:44.508959+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:46:44.508959+0000","last_undegraded":"2024-09-11T14:46:44.508959+0000","last_fullsized":"2024-09-11T14:46:44.508959+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":89,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.523820+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:46:44.523820+0000","last_peered":"2024-09-11T14:46:44.523820+0000","last_clean":"2024-09-11T14:46:44.523820+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:46:44.523820+0000","last_undegraded":"2024-09-11T14:46:44.523820+0000","last_fullsized":"2024-09-11T14:46:44.523820+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.514162+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:46:44.514162+0000","last_peered":"2024-09-11T14:46:44.514162+0000","last_clean":"2024-09-11T14:46:44.514162+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:46:44.514162+0000","last_undegraded":"2024-09-11T14:46:44.514162+0000","last_fullsized":"2024-09-11T14:46:44.514162+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.521574+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:46:44.521574+0000","last_peered":"2024-09-11T14:46:44.521574+0000","last_clean":"2024-09-11T14:46:44.521574+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:46:44.521574+0000","last_undegraded":"2024-09-11T14:46:44.521574+0000","last_fullsized":"2024-09-11T14:46:44.521574+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":90,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.514411+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:46:44.514411+0000","last_peered":"2024-09-11T14:46:44.514411+0000","last_clean":"2024-09-11T14:46:44.514411+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:46:44.514411+0000","last_undegraded":"2024-09-11T14:46:44.514411+0000","last_fullsized":"2024-09-11T14:46:44.514411+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.512373+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:46:44.512373+0000","last_peered":"2024-09-11T14:46:44.512373+0000","last_clean":"2024-09-11T14:46:44.512373+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:46:44.512373+0000","last_undegraded":"2024-09-11T14:46:44.512373+0000","last_fullsized":"2024-09-11T14:46:44.512373+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.510313+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:46:44.510313+0000","last_peered":"2024-09-11T14:46:44.510313+0000","last_clean":"2024-09-11T14:46:44.510313+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:46:44.510313+0000","last_undegraded":"2024-09-11T14:46:44.510313+0000","last_fullsized":"2024-09-11T14:46:44.510313+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.512782+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:46:44.512782+0000","last_peered":"2024-09-11T14:46:44.512782+0000","last_clean":"2024-09-11T14:46:44.512782+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:46:44.512782+0000","last_undegraded":"2024-09-11T14:46:44.512782+0000","last_fullsized":"2024-09-11T14:46:44.512782+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.523616+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:46:44.523616+0000","last_peered":"2024-09-11T14:46:44.523616+0000","last_clean":"2024-09-11T14:46:44.523616+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:46:44.523616+0000","last_undegraded":"2024-09-11T14:46:44.523616+0000","last_fullsized":"2024-09-11T14:46:44.523616+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214202736640,"internally_reserved":0,"allocated":8757248,"data_stored":8757248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53550514176,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673299,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2380,"kb_used_data":2380,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295348,"statfs":{"total":53552873472,"available":53550436352,"internally_reserved":0,"allocated":2437120,"data_stored":2437120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654706004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2304,"kb_used_data":2304,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295424,"statfs":{"total":53552873472,"available":53550514176,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1536,"kb_used_data":1536,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296192,"statfs":{"total":53552873472,"available":53551300608,"internally_reserved":0,"allocated":1572864,"data_stored":1572864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706003,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2308,"kb_used_data":2308,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295420,"statfs":{"total":53552873472,"available":53550510080,"internally_reserved":0,"allocated":2363392,"data_stored":2363392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53550514176,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53550485504,"internally_reserved":0,"allocated":2387968,"data_stored":2387968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551300608,"internally_reserved":0,"allocated":1572864,"data_stored":1572864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53550436352,"internally_reserved":0,"allocated":2437120,"data_stored":2437120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53550514176,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:46:49.130 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-11T14:46:49.130 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-11T14:46:49.240 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:49.232+0000 7fb9c3d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:49.244 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:49.243+0000 7fb9c3d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:49.446 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-11T14:46:49.455 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-11T14:46:49.455 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:46:49.563 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:49.558+0000 7f8cef591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:49.567 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:49.565+0000 7f8cef591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:49.749 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:46:49.750 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:46:49.762 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-09-11T14:46:48.783293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":8528,"kb_used_data":8528,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209182384,"statfs":{"total":214211493888,"available":214202761216,"internally_reserved":0,"allocated":8732672,"data_stored":8732672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.391996"},"pg_stats":[{"pgid":"1.0","version":"13'52","reported_seq":160,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.508959+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:46:44.508959+0000","last_peered":"2024-09-11T14:46:44.508959+0000","last_clean":"2024-09-11T14:46:44.508959+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:46:44.508959+0000","last_undegraded":"2024-09-11T14:46:44.508959+0000","last_fullsized":"2024-09-11T14:46:44.508959+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":89,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.523820+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:46:44.523820+0000","last_peered":"2024-09-11T14:46:44.523820+0000","last_clean":"2024-09-11T14:46:44.523820+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:46:44.523820+0000","last_undegraded":"2024-09-11T14:46:44.523820+0000","last_fullsized":"2024-09-11T14:46:44.523820+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.514162+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:46:44.514162+0000","last_peered":"2024-09-11T14:46:44.514162+0000","last_clean":"2024-09-11T14:46:44.514162+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:46:44.514162+0000","last_undegraded":"2024-09-11T14:46:44.514162+0000","last_fullsized":"2024-09-11T14:46:44.514162+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.521574+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:46:44.521574+0000","last_peered":"2024-09-11T14:46:44.521574+0000","last_clean":"2024-09-11T14:46:44.521574+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:46:44.521574+0000","last_undegraded":"2024-09-11T14:46:44.521574+0000","last_fullsized":"2024-09-11T14:46:44.521574+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":90,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.514411+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:46:44.514411+0000","last_peered":"2024-09-11T14:46:44.514411+0000","last_clean":"2024-09-11T14:46:44.514411+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:46:44.514411+0000","last_undegraded":"2024-09-11T14:46:44.514411+0000","last_fullsized":"2024-09-11T14:46:44.514411+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.512373+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:46:44.512373+0000","last_peered":"2024-09-11T14:46:44.512373+0000","last_clean":"2024-09-11T14:46:44.512373+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:46:44.512373+0000","last_undegraded":"2024-09-11T14:46:44.512373+0000","last_fullsized":"2024-09-11T14:46:44.512373+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.510313+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:46:44.510313+0000","last_peered":"2024-09-11T14:46:44.510313+0000","last_clean":"2024-09-11T14:46:44.510313+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:46:44.510313+0000","last_undegraded":"2024-09-11T14:46:44.510313+0000","last_fullsized":"2024-09-11T14:46:44.510313+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.512782+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:46:44.512782+0000","last_peered":"2024-09-11T14:46:44.512782+0000","last_clean":"2024-09-11T14:46:44.512782+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:46:44.512782+0000","last_undegraded":"2024-09-11T14:46:44.512782+0000","last_fullsized":"2024-09-11T14:46:44.512782+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.523616+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:46:44.523616+0000","last_peered":"2024-09-11T14:46:44.523616+0000","last_clean":"2024-09-11T14:46:44.523616+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:46:44.523616+0000","last_undegraded":"2024-09-11T14:46:44.523616+0000","last_fullsized":"2024-09-11T14:46:44.523616+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214202736640,"internally_reserved":0,"allocated":8757248,"data_stored":8757248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53550514176,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673299,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2380,"kb_used_data":2380,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295348,"statfs":{"total":53552873472,"available":53550436352,"internally_reserved":0,"allocated":2437120,"data_stored":2437120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654706004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2304,"kb_used_data":2304,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295424,"statfs":{"total":53552873472,"available":53550514176,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1536,"kb_used_data":1536,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296192,"statfs":{"total":53552873472,"available":53551300608,"internally_reserved":0,"allocated":1572864,"data_stored":1572864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706003,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2308,"kb_used_data":2308,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295420,"statfs":{"total":53552873472,"available":53550510080,"internally_reserved":0,"allocated":2363392,"data_stored":2363392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53550514176,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53550485504,"internally_reserved":0,"allocated":2387968,"data_stored":2387968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551300608,"internally_reserved":0,"allocated":1572864,"data_stored":1572864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53550436352,"internally_reserved":0,"allocated":2437120,"data_stored":2437120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53550514176,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:46:49.763 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:46:49.872 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:49.867+0000 7fd23c246640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:49.876 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:49.875+0000 7fd23c246640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:50.059 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:46:50.059 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:46:50.072 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-09-11T14:46:48.783293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":8528,"kb_used_data":8528,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209182384,"statfs":{"total":214211493888,"available":214202761216,"internally_reserved":0,"allocated":8732672,"data_stored":8732672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.391996"},"pg_stats":[{"pgid":"1.0","version":"13'52","reported_seq":160,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.508959+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:46:44.508959+0000","last_peered":"2024-09-11T14:46:44.508959+0000","last_clean":"2024-09-11T14:46:44.508959+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:46:44.508959+0000","last_undegraded":"2024-09-11T14:46:44.508959+0000","last_fullsized":"2024-09-11T14:46:44.508959+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":89,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.523820+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:46:44.523820+0000","last_peered":"2024-09-11T14:46:44.523820+0000","last_clean":"2024-09-11T14:46:44.523820+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:46:44.523820+0000","last_undegraded":"2024-09-11T14:46:44.523820+0000","last_fullsized":"2024-09-11T14:46:44.523820+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.514162+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:46:44.514162+0000","last_peered":"2024-09-11T14:46:44.514162+0000","last_clean":"2024-09-11T14:46:44.514162+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:46:44.514162+0000","last_undegraded":"2024-09-11T14:46:44.514162+0000","last_fullsized":"2024-09-11T14:46:44.514162+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.521574+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:46:44.521574+0000","last_peered":"2024-09-11T14:46:44.521574+0000","last_clean":"2024-09-11T14:46:44.521574+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:46:44.521574+0000","last_undegraded":"2024-09-11T14:46:44.521574+0000","last_fullsized":"2024-09-11T14:46:44.521574+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":90,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.514411+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:46:44.514411+0000","last_peered":"2024-09-11T14:46:44.514411+0000","last_clean":"2024-09-11T14:46:44.514411+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:46:44.514411+0000","last_undegraded":"2024-09-11T14:46:44.514411+0000","last_fullsized":"2024-09-11T14:46:44.514411+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.512373+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:46:44.512373+0000","last_peered":"2024-09-11T14:46:44.512373+0000","last_clean":"2024-09-11T14:46:44.512373+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:46:44.512373+0000","last_undegraded":"2024-09-11T14:46:44.512373+0000","last_fullsized":"2024-09-11T14:46:44.512373+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.510313+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:46:44.510313+0000","last_peered":"2024-09-11T14:46:44.510313+0000","last_clean":"2024-09-11T14:46:44.510313+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:46:44.510313+0000","last_undegraded":"2024-09-11T14:46:44.510313+0000","last_fullsized":"2024-09-11T14:46:44.510313+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.512782+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:46:44.512782+0000","last_peered":"2024-09-11T14:46:44.512782+0000","last_clean":"2024-09-11T14:46:44.512782+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:46:44.512782+0000","last_undegraded":"2024-09-11T14:46:44.512782+0000","last_fullsized":"2024-09-11T14:46:44.512782+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.523616+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:46:44.523616+0000","last_peered":"2024-09-11T14:46:44.523616+0000","last_clean":"2024-09-11T14:46:44.523616+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:46:44.523616+0000","last_undegraded":"2024-09-11T14:46:44.523616+0000","last_fullsized":"2024-09-11T14:46:44.523616+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214202736640,"internally_reserved":0,"allocated":8757248,"data_stored":8757248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53550514176,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673299,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2380,"kb_used_data":2380,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295348,"statfs":{"total":53552873472,"available":53550436352,"internally_reserved":0,"allocated":2437120,"data_stored":2437120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654706004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2304,"kb_used_data":2304,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295424,"statfs":{"total":53552873472,"available":53550514176,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1536,"kb_used_data":1536,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296192,"statfs":{"total":53552873472,"available":53551300608,"internally_reserved":0,"allocated":1572864,"data_stored":1572864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706003,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2308,"kb_used_data":2308,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295420,"statfs":{"total":53552873472,"available":53550510080,"internally_reserved":0,"allocated":2363392,"data_stored":2363392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53550514176,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53550485504,"internally_reserved":0,"allocated":2387968,"data_stored":2387968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551300608,"internally_reserved":0,"allocated":1572864,"data_stored":1572864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53550436352,"internally_reserved":0,"allocated":2437120,"data_stored":2437120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53550514176,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:46:50.073 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-11T14:46:50.073 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:46:50.180 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:50.175+0000 7f84b5191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:50.183 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:50.182+0000 7f84b5191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:50.368 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:46:50.368 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:46:50.381 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-09-11T14:46:48.783293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":8528,"kb_used_data":8528,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209182384,"statfs":{"total":214211493888,"available":214202761216,"internally_reserved":0,"allocated":8732672,"data_stored":8732672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.391996"},"pg_stats":[{"pgid":"1.0","version":"13'52","reported_seq":160,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.508959+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:46:44.508959+0000","last_peered":"2024-09-11T14:46:44.508959+0000","last_clean":"2024-09-11T14:46:44.508959+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:46:44.508959+0000","last_undegraded":"2024-09-11T14:46:44.508959+0000","last_fullsized":"2024-09-11T14:46:44.508959+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":89,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.523820+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:46:44.523820+0000","last_peered":"2024-09-11T14:46:44.523820+0000","last_clean":"2024-09-11T14:46:44.523820+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:46:44.523820+0000","last_undegraded":"2024-09-11T14:46:44.523820+0000","last_fullsized":"2024-09-11T14:46:44.523820+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.514162+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:46:44.514162+0000","last_peered":"2024-09-11T14:46:44.514162+0000","last_clean":"2024-09-11T14:46:44.514162+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:46:44.514162+0000","last_undegraded":"2024-09-11T14:46:44.514162+0000","last_fullsized":"2024-09-11T14:46:44.514162+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.521574+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:46:44.521574+0000","last_peered":"2024-09-11T14:46:44.521574+0000","last_clean":"2024-09-11T14:46:44.521574+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:46:44.521574+0000","last_undegraded":"2024-09-11T14:46:44.521574+0000","last_fullsized":"2024-09-11T14:46:44.521574+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":90,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.514411+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:46:44.514411+0000","last_peered":"2024-09-11T14:46:44.514411+0000","last_clean":"2024-09-11T14:46:44.514411+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:46:44.514411+0000","last_undegraded":"2024-09-11T14:46:44.514411+0000","last_fullsized":"2024-09-11T14:46:44.514411+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.512373+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:46:44.512373+0000","last_peered":"2024-09-11T14:46:44.512373+0000","last_clean":"2024-09-11T14:46:44.512373+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:46:44.512373+0000","last_undegraded":"2024-09-11T14:46:44.512373+0000","last_fullsized":"2024-09-11T14:46:44.512373+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.510313+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:46:44.510313+0000","last_peered":"2024-09-11T14:46:44.510313+0000","last_clean":"2024-09-11T14:46:44.510313+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:46:44.510313+0000","last_undegraded":"2024-09-11T14:46:44.510313+0000","last_fullsized":"2024-09-11T14:46:44.510313+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.512782+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:46:44.512782+0000","last_peered":"2024-09-11T14:46:44.512782+0000","last_clean":"2024-09-11T14:46:44.512782+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:46:44.512782+0000","last_undegraded":"2024-09-11T14:46:44.512782+0000","last_fullsized":"2024-09-11T14:46:44.512782+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.523616+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:46:44.523616+0000","last_peered":"2024-09-11T14:46:44.523616+0000","last_clean":"2024-09-11T14:46:44.523616+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:46:44.523616+0000","last_undegraded":"2024-09-11T14:46:44.523616+0000","last_fullsized":"2024-09-11T14:46:44.523616+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214202736640,"internally_reserved":0,"allocated":8757248,"data_stored":8757248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53550514176,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673299,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2380,"kb_used_data":2380,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295348,"statfs":{"total":53552873472,"available":53550436352,"internally_reserved":0,"allocated":2437120,"data_stored":2437120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654706004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2304,"kb_used_data":2304,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295424,"statfs":{"total":53552873472,"available":53550514176,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1536,"kb_used_data":1536,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296192,"statfs":{"total":53552873472,"available":53551300608,"internally_reserved":0,"allocated":1572864,"data_stored":1572864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706003,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2308,"kb_used_data":2308,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295420,"statfs":{"total":53552873472,"available":53550510080,"internally_reserved":0,"allocated":2363392,"data_stored":2363392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53550514176,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53550485504,"internally_reserved":0,"allocated":2387968,"data_stored":2387968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551300608,"internally_reserved":0,"allocated":1572864,"data_stored":1572864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53550436352,"internally_reserved":0,"allocated":2437120,"data_stored":2437120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53550514176,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:46:50.381 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T14:46:50.490 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:50.484+0000 7f62cd591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:50.493 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:50.492+0000 7f62cd591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:50.680 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:46:50.680 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":40,"fsid":"0f7e90de-623e-43a2-a836-b493e06f03fc","created":"2024-09-11T14:40:58.371084+0000","modified":"2024-09-11T14:46:44.471423+0000","last_up_change":"2024-09-11T14:41:05.766940+0000","last_in_change":"2024-09-11T14:41:00.494637+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T14:41:06.733924+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T14:41:08.627066+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"05592c04-44ae-46d8-bcf7-a865d8397d5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":920133847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6803","nonce":920133847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6809","nonce":920133847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6807","nonce":920133847}]},"public_addr":"172.21.15.181:6804/920133847","cluster_addr":"172.21.15.181:6803/920133847","heartbeat_back_addr":"172.21.15.181:6809/920133847","heartbeat_front_addr":"172.21.15.181:6807/920133847","state":["exists","up"]},{"osd":1,"uuid":"70bdeffe-ad6e-4417-92f7-6021f541c690","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6801","nonce":3969919471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3969919471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3969919471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6819","nonce":3969919471}]},"public_addr":"172.21.15.181:6801/3969919471","cluster_addr":"172.21.15.181:6802/3969919471","heartbeat_back_addr":"172.21.15.181:6820/3969919471","heartbeat_front_addr":"172.21.15.181:6819/3969919471","state":["exists","up"]},{"osd":2,"uuid":"d3fd0624-0127-4f26-ba89-ca34a2c562ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6811","nonce":692234911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":692234911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":692234911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":692234911}]},"public_addr":"172.21.15.203:6811/692234911","cluster_addr":"172.21.15.203:6810/692234911","heartbeat_back_addr":"172.21.15.203:6808/692234911","heartbeat_front_addr":"172.21.15.203:6809/692234911","state":["exists","up"]},{"osd":3,"uuid":"24bbbdd2-a931-4cc2-a24d-61d131ece2c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6803","nonce":667507199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":667507199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":667507199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6805","nonce":667507199}]},"public_addr":"172.21.15.203:6803/667507199","cluster_addr":"172.21.15.203:6802/667507199","heartbeat_back_addr":"172.21.15.203:6804/667507199","heartbeat_front_addr":"172.21.15.203:6805/667507199","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T14:46:51.695 INFO:tasks.ceph:Scrubbing osd.0 2024-09-11T14:46:51.696 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-11T14:46:51.822 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:51.812+0000 7f872f191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:51.825 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:51.824+0000 7f872f191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:51.840 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T14:46:51.840 INFO:teuthology.orchestra.run.smithi181.stdout: "success": "" 2024-09-11T14:46:51.840 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T14:46:51.851 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-11T14:46:51.961 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:51.954+0000 7fb73f6a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:51.965 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:51.963+0000 7fb73f6a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:52.148 INFO:teuthology.orchestra.run.smithi181.stderr:instructed osd(s) 0 to deep-scrub 2024-09-11T14:46:52.166 INFO:tasks.ceph:Scrubbing osd.1 2024-09-11T14:46:52.166 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-11T14:46:52.281 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:52.271+0000 7fab69e88640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:52.285 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:52.284+0000 7fab69e88640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:52.301 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T14:46:52.301 INFO:teuthology.orchestra.run.smithi181.stdout: "success": "" 2024-09-11T14:46:52.301 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T14:46:52.312 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-11T14:46:52.421 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:52.414+0000 7f9361a5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:52.424 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:52.423+0000 7f9361a5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:52.612 INFO:teuthology.orchestra.run.smithi181.stderr:instructed osd(s) 1 to deep-scrub 2024-09-11T14:46:52.629 INFO:tasks.ceph:Scrubbing osd.2 2024-09-11T14:46:52.630 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-11T14:46:52.743 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:52.732+0000 7f0330351640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:52.746 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:52.745+0000 7f0330351640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:52.764 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T14:46:52.764 INFO:teuthology.orchestra.run.smithi181.stdout: "success": "" 2024-09-11T14:46:52.764 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T14:46:52.775 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-11T14:46:52.885 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:52.878+0000 7f329146c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:52.888 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:52.887+0000 7f329146c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:53.073 INFO:teuthology.orchestra.run.smithi181.stderr:instructed osd(s) 2 to deep-scrub 2024-09-11T14:46:53.089 INFO:tasks.ceph:Scrubbing osd.3 2024-09-11T14:46:53.089 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-11T14:46:53.201 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:53.191+0000 7fc9350ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:53.205 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:53.203+0000 7fc9350ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:53.222 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-11T14:46:53.222 INFO:teuthology.orchestra.run.smithi181.stdout: "success": "" 2024-09-11T14:46:53.222 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-11T14:46:53.233 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-11T14:46:53.343 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:53.336+0000 7fe098951640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:53.347 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:53.346+0000 7fe098951640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:53.535 INFO:teuthology.orchestra.run.smithi181.stderr:instructed osd(s) 3 to deep-scrub 2024-09-11T14:46:53.548 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:46:53.659 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:53.653+0000 7fbe11791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:53.662 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:46:53.661+0000 7fbe11791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:46:53.849 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:46:53.849 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:46:53.862 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2024-09-11T14:46:52.784292+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":8552,"kb_used_data":8552,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209182360,"statfs":{"total":214211493888,"available":214202736640,"internally_reserved":0,"allocated":8757248,"data_stored":8757248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.386431"},"pg_stats":[{"pgid":"1.0","version":"13'52","reported_seq":160,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.508959+0000","last_change":"2024-09-11T14:41:07.860610+0000","last_active":"2024-09-11T14:46:44.508959+0000","last_peered":"2024-09-11T14:46:44.508959+0000","last_clean":"2024-09-11T14:46:44.508959+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:46:44.508959+0000","last_undegraded":"2024-09-11T14:46:44.508959+0000","last_fullsized":"2024-09-11T14:46:44.508959+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:06.772128+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":89,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.523820+0000","last_change":"2024-09-11T14:41:11.832857+0000","last_active":"2024-09-11T14:46:44.523820+0000","last_peered":"2024-09-11T14:46:44.523820+0000","last_clean":"2024-09-11T14:46:44.523820+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:46:44.523820+0000","last_undegraded":"2024-09-11T14:46:44.523820+0000","last_fullsized":"2024-09-11T14:46:44.523820+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.514162+0000","last_change":"2024-09-11T14:41:11.829252+0000","last_active":"2024-09-11T14:46:44.514162+0000","last_peered":"2024-09-11T14:46:44.514162+0000","last_clean":"2024-09-11T14:46:44.514162+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:46:44.514162+0000","last_undegraded":"2024-09-11T14:46:44.514162+0000","last_fullsized":"2024-09-11T14:46:44.514162+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.521574+0000","last_change":"2024-09-11T14:41:11.826815+0000","last_active":"2024-09-11T14:46:44.521574+0000","last_peered":"2024-09-11T14:46:44.521574+0000","last_clean":"2024-09-11T14:46:44.521574+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:46:44.521574+0000","last_undegraded":"2024-09-11T14:46:44.521574+0000","last_fullsized":"2024-09-11T14:46:44.521574+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":90,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.514411+0000","last_change":"2024-09-11T14:41:11.827914+0000","last_active":"2024-09-11T14:46:44.514411+0000","last_peered":"2024-09-11T14:46:44.514411+0000","last_clean":"2024-09-11T14:46:44.514411+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:46:44.514411+0000","last_undegraded":"2024-09-11T14:46:44.514411+0000","last_fullsized":"2024-09-11T14:46:44.514411+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.512373+0000","last_change":"2024-09-11T14:41:11.832919+0000","last_active":"2024-09-11T14:46:44.512373+0000","last_peered":"2024-09-11T14:46:44.512373+0000","last_clean":"2024-09-11T14:46:44.512373+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:46:44.512373+0000","last_undegraded":"2024-09-11T14:46:44.512373+0000","last_fullsized":"2024-09-11T14:46:44.512373+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.510313+0000","last_change":"2024-09-11T14:41:11.830457+0000","last_active":"2024-09-11T14:46:44.510313+0000","last_peered":"2024-09-11T14:46:44.510313+0000","last_clean":"2024-09-11T14:46:44.510313+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:46:44.510313+0000","last_undegraded":"2024-09-11T14:46:44.510313+0000","last_fullsized":"2024-09-11T14:46:44.510313+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.512782+0000","last_change":"2024-09-11T14:41:11.832314+0000","last_active":"2024-09-11T14:46:44.512782+0000","last_peered":"2024-09-11T14:46:44.512782+0000","last_clean":"2024-09-11T14:46:44.512782+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:46:44.512782+0000","last_undegraded":"2024-09-11T14:46:44.512782+0000","last_fullsized":"2024-09-11T14:46:44.512782+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:44.523616+0000","last_change":"2024-09-11T14:41:11.832784+0000","last_active":"2024-09-11T14:46:44.523616+0000","last_peered":"2024-09-11T14:46:44.523616+0000","last_clean":"2024-09-11T14:46:44.523616+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:46:44.523616+0000","last_undegraded":"2024-09-11T14:46:44.523616+0000","last_fullsized":"2024-09-11T14:46:44.523616+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:41:08.780496+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214202736640,"internally_reserved":0,"allocated":8757248,"data_stored":8757248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53550514176,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673304,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2380,"kb_used_data":2380,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295348,"statfs":{"total":53552873472,"available":53550436352,"internally_reserved":0,"allocated":2437120,"data_stored":2437120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654706008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2304,"kb_used_data":2304,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295424,"statfs":{"total":53552873472,"available":53550514176,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1536,"kb_used_data":1536,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296192,"statfs":{"total":53552873472,"available":53551300608,"internally_reserved":0,"allocated":1572864,"data_stored":1572864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2332,"kb_used_data":2332,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295396,"statfs":{"total":53552873472,"available":53550485504,"internally_reserved":0,"allocated":2387968,"data_stored":2387968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53550514176,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53550485504,"internally_reserved":0,"allocated":2387968,"data_stored":2387968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551300608,"internally_reserved":0,"allocated":1572864,"data_stored":1572864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53550436352,"internally_reserved":0,"allocated":2437120,"data_stored":2437120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53550514176,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:46:53.865 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-11T14:41:06.772128+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=41, tm_sec=6, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=46, tm_sec=50, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:46:53.865 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-11T14:41:08.780496+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=41, tm_sec=8, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=46, tm_sec=50, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:46:53.866 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-11T14:41:08.780496+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=41, tm_sec=8, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=46, tm_sec=50, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:46:53.866 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-11T14:41:08.780496+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=41, tm_sec=8, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=46, tm_sec=50, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:46:53.866 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-11T14:41:08.780496+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=41, tm_sec=8, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=46, tm_sec=50, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:46:53.866 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-11T14:41:08.780496+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=41, tm_sec=8, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=46, tm_sec=50, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:46:53.866 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-11T14:41:08.780496+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=41, tm_sec=8, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=46, tm_sec=50, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:46:53.866 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-11T14:41:08.780496+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=41, tm_sec=8, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=46, tm_sec=50, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:46:53.867 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-11T14:41:08.780496+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=41, tm_sec=8, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=46, tm_sec=50, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:46:53.867 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-11T14:47:13.867 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:47:13.989 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:47:13.984+0000 7f4aa8f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:47:13.993 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:47:13.992+0000 7f4aa8f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:47:14.176 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-11T14:47:14.176 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-11T14:47:14.188 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":217,"stamp":"2024-09-11T14:47:12.789223+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":8552,"kb_used_data":8552,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209182360,"statfs":{"total":214211493888,"available":214202736640,"internally_reserved":0,"allocated":8757248,"data_stored":8757248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002788"},"pg_stats":[{"pgid":"1.0","version":"13'52","reported_seq":162,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:53.536678+0000","last_change":"2024-09-11T14:46:53.598718+0000","last_active":"2024-09-11T14:46:53.536678+0000","last_peered":"2024-09-11T14:46:53.536678+0000","last_clean":"2024-09-11T14:46:53.536678+0000","last_became_active":"2024-09-11T14:41:07.839217+0000","last_became_peered":"2024-09-11T14:41:07.839217+0000","last_unstale":"2024-09-11T14:46:53.536678+0000","last_undegraded":"2024-09-11T14:46:53.536678+0000","last_fullsized":"2024-09-11T14:46:53.536678+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"13'52","last_scrub_stamp":"2024-09-11T14:46:53.598491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:46:53.598491+0000","last_clean_scrub_stamp":"2024-09-11T14:41:06.772128+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":91,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:52.614947+0000","last_change":"2024-09-11T14:46:52.643240+0000","last_active":"2024-09-11T14:46:52.614947+0000","last_peered":"2024-09-11T14:46:52.614947+0000","last_clean":"2024-09-11T14:46:52.614947+0000","last_became_active":"2024-09-11T14:41:09.914102+0000","last_became_peered":"2024-09-11T14:41:09.914102+0000","last_unstale":"2024-09-11T14:46:52.614947+0000","last_undegraded":"2024-09-11T14:46:52.614947+0000","last_fullsized":"2024-09-11T14:46:52.614947+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"14'1","last_scrub_stamp":"2024-09-11T14:46:52.643035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:46:52.643035+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":90,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:52.150047+0000","last_change":"2024-09-11T14:46:52.173328+0000","last_active":"2024-09-11T14:46:52.150047+0000","last_peered":"2024-09-11T14:46:52.150047+0000","last_clean":"2024-09-11T14:46:52.150047+0000","last_became_active":"2024-09-11T14:41:09.932551+0000","last_became_peered":"2024-09-11T14:41:09.932551+0000","last_unstale":"2024-09-11T14:46:52.150047+0000","last_undegraded":"2024-09-11T14:46:52.150047+0000","last_fullsized":"2024-09-11T14:46:52.150047+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:46:52.173149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:46:52.173149+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":90,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:52.612985+0000","last_change":"2024-09-11T14:46:52.632572+0000","last_active":"2024-09-11T14:46:52.612985+0000","last_peered":"2024-09-11T14:46:52.612985+0000","last_clean":"2024-09-11T14:46:52.612985+0000","last_became_active":"2024-09-11T14:41:09.918327+0000","last_became_peered":"2024-09-11T14:41:09.918327+0000","last_unstale":"2024-09-11T14:46:52.612985+0000","last_undegraded":"2024-09-11T14:46:52.612985+0000","last_fullsized":"2024-09-11T14:46:52.612985+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:46:52.632400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:46:52.632400+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":92,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:52.150048+0000","last_change":"2024-09-11T14:46:52.184451+0000","last_active":"2024-09-11T14:46:52.150048+0000","last_peered":"2024-09-11T14:46:52.150048+0000","last_clean":"2024-09-11T14:46:52.150048+0000","last_became_active":"2024-09-11T14:41:09.945679+0000","last_became_peered":"2024-09-11T14:41:09.945679+0000","last_unstale":"2024-09-11T14:46:52.150048+0000","last_undegraded":"2024-09-11T14:46:52.150048+0000","last_fullsized":"2024-09-11T14:46:52.150048+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"16'2","last_scrub_stamp":"2024-09-11T14:46:52.184106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:46:52.184106+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":90,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:53.075566+0000","last_change":"2024-09-11T14:46:53.091911+0000","last_active":"2024-09-11T14:46:53.075566+0000","last_peered":"2024-09-11T14:46:53.075566+0000","last_clean":"2024-09-11T14:46:53.075566+0000","last_became_active":"2024-09-11T14:41:09.904403+0000","last_became_peered":"2024-09-11T14:41:09.904403+0000","last_unstale":"2024-09-11T14:46:53.075566+0000","last_undegraded":"2024-09-11T14:46:53.075566+0000","last_fullsized":"2024-09-11T14:46:53.075566+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:46:53.091680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:46:53.091680+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":90,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:53.538657+0000","last_change":"2024-09-11T14:46:53.564440+0000","last_active":"2024-09-11T14:46:53.538657+0000","last_peered":"2024-09-11T14:46:53.538657+0000","last_clean":"2024-09-11T14:46:53.538657+0000","last_became_active":"2024-09-11T14:41:09.887518+0000","last_became_peered":"2024-09-11T14:41:09.887518+0000","last_unstale":"2024-09-11T14:46:53.538657+0000","last_undegraded":"2024-09-11T14:46:53.538657+0000","last_fullsized":"2024-09-11T14:46:53.538657+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:46:53.564224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:46:53.564224+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":90,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:53.075576+0000","last_change":"2024-09-11T14:46:53.093419+0000","last_active":"2024-09-11T14:46:53.075576+0000","last_peered":"2024-09-11T14:46:53.075576+0000","last_clean":"2024-09-11T14:46:53.075576+0000","last_became_active":"2024-09-11T14:41:09.931169+0000","last_became_peered":"2024-09-11T14:41:09.931169+0000","last_unstale":"2024-09-11T14:46:53.075576+0000","last_undegraded":"2024-09-11T14:46:53.075576+0000","last_fullsized":"2024-09-11T14:46:53.075576+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:46:53.093224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:46:53.093224+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":90,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T14:46:52.616056+0000","last_change":"2024-09-11T14:46:52.635937+0000","last_active":"2024-09-11T14:46:52.616056+0000","last_peered":"2024-09-11T14:46:52.616056+0000","last_clean":"2024-09-11T14:46:52.616056+0000","last_became_active":"2024-09-11T14:41:09.928596+0000","last_became_peered":"2024-09-11T14:41:09.928596+0000","last_unstale":"2024-09-11T14:46:52.616056+0000","last_undegraded":"2024-09-11T14:46:52.616056+0000","last_fullsized":"2024-09-11T14:46:52.616056+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:46:52.635738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:46:52.635738+0000","last_clean_scrub_stamp":"2024-09-11T14:41:08.780496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214202736640,"internally_reserved":0,"allocated":8757248,"data_stored":8757248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53550514176,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673324,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2380,"kb_used_data":2380,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295348,"statfs":{"total":53552873472,"available":53550436352,"internally_reserved":0,"allocated":2437120,"data_stored":2437120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654706028,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2304,"kb_used_data":2304,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295424,"statfs":{"total":53552873472,"available":53550514176,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706028,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1536,"kb_used_data":1536,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296192,"statfs":{"total":53552873472,"available":53551300608,"internally_reserved":0,"allocated":1572864,"data_stored":1572864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706028,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2332,"kb_used_data":2332,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295396,"statfs":{"total":53552873472,"available":53550485504,"internally_reserved":0,"allocated":2387968,"data_stored":2387968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53550514176,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53550485504,"internally_reserved":0,"allocated":2387968,"data_stored":2387968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551300608,"internally_reserved":0,"allocated":1572864,"data_stored":1572864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53550436352,"internally_reserved":0,"allocated":2437120,"data_stored":2437120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53550514176,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:47:14.190 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-11T14:47:14.298 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:47:14.289+0000 7f73d0b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:47:14.302 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-11T14:47:14.300+0000 7f73d0b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:47:14.505 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-11T14:47:14.506 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-11T14:47:14.506 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-11T14:47:14.506 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T14:47:14.579 INFO:tasks.ceph.osd.0:Stopped 2024-09-11T14:47:14.579 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-11T14:47:14.580 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T14:47:14.665 INFO:tasks.ceph.osd.1:Stopped 2024-09-11T14:47:14.666 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-11T14:47:14.666 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T14:47:14.749 INFO:tasks.ceph.osd.2:Stopped 2024-09-11T14:47:14.749 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-11T14:47:14.749 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T14:47:14.831 INFO:tasks.ceph.osd.3:Stopped 2024-09-11T14:47:14.831 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-11T14:47:14.831 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-11T14:47:14.831 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T14:47:14.861 INFO:tasks.ceph.mgr.x:Stopped 2024-09-11T14:47:14.862 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-11T14:47:14.862 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-11T14:47:14.862 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T14:47:14.874 INFO:tasks.ceph.mon.a:Stopped 2024-09-11T14:47:14.874 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-11T14:47:14.874 DEBUG:teuthology.orchestra.run.smithi181:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v SLOW_OPS | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-09-11T14:47:14.947 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-11T14:47:14.947 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-11T14:47:15.129 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-11T14:47:15.130 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-11T14:47:15.217 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi203.front.sepia.ceph.com 2024-09-11T14:47:15.217 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-11T14:47:15.353 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi203.front.sepia.ceph.com 2024-09-11T14:47:15.353 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-11T14:47:15.433 INFO:tasks.ceph:Archiving mon data... 2024-09-11T14:47:15.434 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901371/data/mon.a.tgz 2024-09-11T14:47:15.434 DEBUG:teuthology.orchestra.run.smithi181:> mktemp 2024-09-11T14:47:15.450 INFO:teuthology.orchestra.run.smithi181.stdout:/tmp/tmp.Bk1WEqFuUn 2024-09-11T14:47:15.450 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Bk1WEqFuUn 2024-09-11T14:47:15.617 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0666 /tmp/tmp.Bk1WEqFuUn 2024-09-11T14:47:15.700 DEBUG:teuthology.orchestra.remote:smithi181:/tmp/tmp.Bk1WEqFuUn is 262KB 2024-09-11T14:47:15.795 DEBUG:teuthology.orchestra.run.smithi181:> rm -fr /tmp/tmp.Bk1WEqFuUn 2024-09-11T14:47:15.810 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-11T14:47:15.810 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T14:47:15.851 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T14:47:15.929 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-11T14:47:15.945 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-11T14:47:15.945 INFO:tasks.ceph:Archiving crash dumps... 2024-09-11T14:47:15.947 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901371/remote/smithi181/crash 2024-09-11T14:47:15.948 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-11T14:47:15.995 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901371/remote/smithi203/crash 2024-09-11T14:47:15.997 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-11T14:47:16.038 INFO:tasks.ceph:Compressing logs... 2024-09-11T14:47:16.070 DEBUG:teuthology.orchestra.run.smithi181:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T14:47:16.073 DEBUG:teuthology.orchestra.run.smithi203:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T14:47:16.094 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-11T14:47:16.095 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32671.log 2024-09-11T14:47:16.144 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27663.log 2024-09-11T14:47:16.144 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27703.log 2024-09-11T14:47:16.144 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.27663.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-11T14:47:16.144 INFO:teuthology.orchestra.run.smithi203.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27663.log.gz 2024-09-11T14:47:16.144 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.27703.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-11T14:47:16.145 INFO:teuthology.orchestra.run.smithi203.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.27703.log.gz 2024-09-11T14:47:16.145 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27613.log 2024-09-11T14:47:16.145 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-11T14:47:16.145 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.27613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27743.log 2024-09-11T14:47:16.145 INFO:teuthology.orchestra.run.smithi203.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27613.log.gz 2024-09-11T14:47:16.145 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.27743.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.27743.log.gz 2024-09-11T14:47:16.191 INFO:teuthology.orchestra.run.smithi203.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-11T14:47:16.211 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36516.log 2024-09-11T14:47:16.211 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30049.log 2024-09-11T14:47:16.211 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32671.log.gz 2024-09-11T14:47:16.211 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39838.log 2024-09-11T14:47:16.211 INFO:teuthology.orchestra.run.smithi181.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36516.log.gz 2024-09-11T14:47:16.211 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.30049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30435.log 2024-09-11T14:47:16.211 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30049.log.gz 2024-09-11T14:47:16.211 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33559.log 2024-09-11T14:47:16.211 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39838.log.gz 2024-09-11T14:47:16.212 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.30435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28652.log 2024-09-11T14:47:16.212 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30435.log.gz 2024-09-11T14:47:16.212 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38346.log 2024-09-11T14:47:16.212 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33559.log.gz 2024-09-11T14:47:16.212 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.28652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42809.log 2024-09-11T14:47:16.212 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28652.log.gz 2024-09-11T14:47:16.212 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42325.log 2024-09-11T14:47:16.212 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38346.log.gz 2024-09-11T14:47:16.212 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31972.log 2024-09-11T14:47:16.212 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42809.log.gz 2024-09-11T14:47:16.212 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31879.log 2024-09-11T14:47:16.212 INFO:teuthology.orchestra.run.smithi181.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42325.log.gz 2024-09-11T14:47:16.212 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.31972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31334.log 2024-09-11T14:47:16.212 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31972.log.gz 2024-09-11T14:47:16.213 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.31879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36794.log 2024-09-11T14:47:16.213 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31879.log.gz 2024-09-11T14:47:16.213 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.31334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34017.log 2024-09-11T14:47:16.213 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31334.log.gz 2024-09-11T14:47:16.213 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34195.log 2024-09-11T14:47:16.213 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36794.log.gz 2024-09-11T14:47:16.213 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28319.log 2024-09-11T14:47:16.213 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34017.log.gz 2024-09-11T14:47:16.213 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29716.log 2024-09-11T14:47:16.213 INFO:teuthology.orchestra.run.smithi181.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.34195.log.gz 2024-09-11T14:47:16.213 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.28319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41001.log 2024-09-11T14:47:16.213 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28319.log.gz 2024-09-11T14:47:16.213 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37221.log 2024-09-11T14:47:16.213 INFO:teuthology.orchestra.run.smithi181.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29716.log.gz 2024-09-11T14:47:16.213 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35359.log 2024-09-11T14:47:16.214 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41001.log.gz 2024-09-11T14:47:16.214 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41507.log 2024-09-11T14:47:16.214 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37221.log.gz 2024-09-11T14:47:16.214 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34544.log 2024-09-11T14:47:16.214 INFO:teuthology.orchestra.run.smithi181.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.35359.log.gz 2024-09-11T14:47:16.214 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42683.log 2024-09-11T14:47:16.214 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41507.log.gz 2024-09-11T14:47:16.214 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31643.log 2024-09-11T14:47:16.214 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34544.log.gz 2024-09-11T14:47:16.214 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29674.log 2024-09-11T14:47:16.214 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42683.log.gz 2024-09-11T14:47:16.214 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.31643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33957.log 2024-09-11T14:47:16.214 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31643.log.gz 2024-09-11T14:47:16.214 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40107.log 2024-09-11T14:47:16.215 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29674.log.gz 2024-09-11T14:47:16.215 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31055.log 2024-09-11T14:47:16.215 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33957.log.gz 2024-09-11T14:47:16.215 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40812.log 2024-09-11T14:47:16.215 INFO:teuthology.orchestra.run.smithi181.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40107.log.gz 2024-09-11T14:47:16.215 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.31055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40247.log 2024-09-11T14:47:16.215 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31055.log.gz 2024-09-11T14:47:16.215 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38957.log 2024-09-11T14:47:16.215 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40812.log.gz 2024-09-11T14:47:16.215 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38444.log 2024-09-11T14:47:16.215 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40247.log.gz 2024-09-11T14:47:16.215 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31503.log 2024-09-11T14:47:16.215 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38957.log.gz 2024-09-11T14:47:16.215 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30009.log 2024-09-11T14:47:16.216 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38444.log.gz 2024-09-11T14:47:16.216 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.31503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31923.log 2024-09-11T14:47:16.216 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31503.log.gz 2024-09-11T14:47:16.216 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.30009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41373.log 2024-09-11T14:47:16.216 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30009.log.gz 2024-09-11T14:47:16.216 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.31923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38619.log 2024-09-11T14:47:16.216 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31923.log.gz 2024-09-11T14:47:16.216 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31245.log 2024-09-11T14:47:16.216 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41373.log.gz 2024-09-11T14:47:16.216 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38577.log 2024-09-11T14:47:16.216 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38619.log.gz 2024-09-11T14:47:16.216 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.31245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39878.log 2024-09-11T14:47:16.216 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31245.log.gz 2024-09-11T14:47:16.217 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40156.log 2024-09-11T14:47:16.217 INFO:teuthology.orchestra.run.smithi181.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.38577.log.gz 2024-09-11T14:47:16.217 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39519.log 2024-09-11T14:47:16.217 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39878.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39878.log.gz 2024-09-11T14:47:16.217 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42641.log 2024-09-11T14:47:16.217 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40156.log.gz 2024-09-11T14:47:16.217 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.30163.log 2024-09-11T14:47:16.217 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39519.log.gz 2024-09-11T14:47:16.217 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40903.log 2024-09-11T14:47:16.217 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42641.log.gz 2024-09-11T14:47:16.217 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.0.30163.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-11T14:47:16.217 INFO:teuthology.orchestra.run.smithi181.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.0.30163.log.gz 2024-09-11T14:47:16.217 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30880.log 2024-09-11T14:47:16.218 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40903.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40903.log.gz 2024-09-11T14:47:16.218 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32909.log 2024-09-11T14:47:16.218 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.30880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38024.log 2024-09-11T14:47:16.218 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30880.log.gz 2024-09-11T14:47:16.218 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30654.log 2024-09-11T14:47:16.218 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32909.log.gz 2024-09-11T14:47:16.218 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29188.log 2024-09-11T14:47:16.218 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38024.log.gz 2024-09-11T14:47:16.218 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.30654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41415.log 2024-09-11T14:47:16.218 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30654.log.gz 2024-09-11T14:47:16.218 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38255.log 2024-09-11T14:47:16.218 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29188.log.gz 2024-09-11T14:47:16.218 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41415.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41415.log.gz 2024-09-11T14:47:16.218 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30274.log 2024-09-11T14:47:16.219 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42302.log 2024-09-11T14:47:16.219 INFO:teuthology.orchestra.run.smithi181.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.38255.log.gz 2024-09-11T14:47:16.219 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.30274.log: 50.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.30274.log.gz 2024-09-11T14:47:16.219 INFO:teuthology.orchestra.run.smithi181.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.36305.log 2024-09-11T14:47:16.219 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35404.log 2024-09-11T14:47:16.219 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42302.log.gz 2024-09-11T14:47:16.219 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31414.log 2024-09-11T14:47:16.219 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36305.log.gz 2024-09-11T14:47:16.219 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35404.log: gzip -5 --verbose -- 50.0% /var/log/ceph/ceph-client.admin.38304.log -- replaced with /var/log/ceph/ceph-client.admin.35404.log.gz 2024-09-11T14:47:16.219 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-11T14:47:16.219 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.31414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35627.log 2024-09-11T14:47:16.219 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31414.log.gz 2024-09-11T14:47:16.219 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29548.log 2024-09-11T14:47:16.220 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38304.log.gz 2024-09-11T14:47:16.220 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33777.log 2024-09-11T14:47:16.220 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35627.log.gz 2024-09-11T14:47:16.220 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29296.log 50.0% 2024-09-11T14:47:16.220 INFO:teuthology.orchestra.run.smithi181.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29548.log.gz 2024-09-11T14:47:16.220 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41986.log 2024-09-11T14:47:16.220 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33777.log.gz 2024-09-11T14:47:16.220 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43271.log 2024-09-11T14:47:16.220 INFO:teuthology.orchestra.run.smithi181.stderr: 48.0% -- replaced with /var/log/ceph/ceph-client.admin.29296.log.gz 2024-09-11T14:47:16.220 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43145.log 2024-09-11T14:47:16.220 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41986.log.gz 2024-09-11T14:47:16.220 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32371.log 2024-09-11T14:47:16.220 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43271.log.gz 2024-09-11T14:47:16.221 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28918.log 2024-09-11T14:47:16.221 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43145.log.gz 2024-09-11T14:47:16.221 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36376.log 2024-09-11T14:47:16.221 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32371.log.gz 2024-09-11T14:47:16.221 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.28918.log: gzip -5 --verbose -- 49.6% /var/log/ceph/ceph-client.admin.37052.log 2024-09-11T14:47:16.221 INFO:teuthology.orchestra.run.smithi181.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28918.log.gz 2024-09-11T14:47:16.221 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40016.log 2024-09-11T14:47:16.221 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36376.log.gz 2024-09-11T14:47:16.221 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32328.log 2024-09-11T14:47:16.221 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37052.log.gz 2024-09-11T14:47:16.221 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40016.log: gzip -5 --verbose -- 49.6% /var/log/ceph/ceph-client.admin.32154.log 2024-09-11T14:47:16.221 INFO:teuthology.orchestra.run.smithi181.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40016.log.gz 2024-09-11T14:47:16.221 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31832.log 2024-09-11T14:47:16.222 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32328.log.gz 2024-09-11T14:47:16.222 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38115.log 2024-09-11T14:47:16.222 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32154.log.gz 2024-09-11T14:47:16.222 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.31832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28708.log 2024-09-11T14:47:16.222 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31832.log.gz 2024-09-11T14:47:16.222 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31374.log 2024-09-11T14:47:16.222 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38115.log.gz 2024-09-11T14:47:16.222 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.28708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30747.log 2024-09-11T14:47:16.222 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28708.log.gz 2024-09-11T14:47:16.222 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.31374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30475.log 2024-09-11T14:47:16.222 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31374.log.gz 2024-09-11T14:47:16.222 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.30747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39310.log 2024-09-11T14:47:16.222 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30747.log.gz 2024-09-11T14:47:16.223 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.30475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31592.log 2024-09-11T14:47:16.223 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30475.log.gz 2024-09-11T14:47:16.223 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41591.log 2024-09-11T14:47:16.223 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39310.log.gz 2024-09-11T14:47:16.223 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.31592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38066.log 2024-09-11T14:47:16.223 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31592.log.gz 2024-09-11T14:47:16.223 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35278.log 2024-09-11T14:47:16.223 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41591.log.gz 2024-09-11T14:47:16.223 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38066.log: gzip -5 50.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.38066.log.gz -- 2024-09-11T14:47:16.223 INFO:teuthology.orchestra.run.smithi181.stderr: /var/log/ceph/ceph-client.admin.41232.log 2024-09-11T14:47:16.223 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29961.log 2024-09-11T14:47:16.223 INFO:teuthology.orchestra.run.smithi181.stderr: 48.0% -- replaced with /var/log/ceph/ceph-client.admin.35278.log.gz 2024-09-11T14:47:16.223 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34624.log 2024-09-11T14:47:16.223 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41232.log.gz 2024-09-11T14:47:16.224 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40198.log 2024-09-11T14:47:16.224 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29961.log.gz 2024-09-11T14:47:16.224 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32451.log 2024-09-11T14:47:16.224 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34624.log.gz 2024-09-11T14:47:16.224 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41675.log 2024-09-11T14:47:16.224 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40198.log.gz 2024-09-11T14:47:16.224 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30787.log 2024-09-11T14:47:16.224 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32451.log.gz 2024-09-11T14:47:16.224 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36185.log 2024-09-11T14:47:16.224 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41675.log.gz 2024-09-11T14:47:16.224 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.30787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38164.log 2024-09-11T14:47:16.224 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30787.log.gz 2024-09-11T14:47:16.224 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32021.log 2024-09-11T14:47:16.225 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36185.log.gz 2024-09-11T14:47:16.225 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39798.log 2024-09-11T14:47:16.225 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38164.log.gz 2024-09-11T14:47:16.225 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40952.log 2024-09-11T14:47:16.225 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32021.log.gz 2024-09-11T14:47:16.225 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38894.log 2024-09-11T14:47:16.225 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39798.log.gz 2024-09-11T14:47:16.225 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40296.log 2024-09-11T14:47:16.225 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40952.log.gz 2024-09-11T14:47:16.225 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40723.log 2024-09-11T14:47:16.225 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38894.log.gz 2024-09-11T14:47:16.225 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39003.log 2024-09-11T14:47:16.225 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40296.log.gz 2024-09-11T14:47:16.226 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33470.log 2024-09-11T14:47:16.226 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40723.log.gz 2024-09-11T14:47:16.226 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39003.log: gzip -5 --verbose -- 49.6% /var/log/ceph/ceph-client.admin.36565.log -- replaced with /var/log/ceph/ceph-client.admin.39003.log.gz 2024-09-11T14:47:16.226 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-11T14:47:16.226 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31195.log 2024-09-11T14:47:16.226 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33470.log.gz 2024-09-11T14:47:16.226 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35725.log 2024-09-11T14:47:16.226 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36565.log.gz 2024-09-11T14:47:16.226 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.31195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33868.log 2024-09-11T14:47:16.226 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31195.log.gz 2024-09-11T14:47:16.226 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29316.log 2024-09-11T14:47:16.226 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35725.log.gz 2024-09-11T14:47:16.226 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41324.log 2024-09-11T14:47:16.227 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33868.log.gz 2024-09-11T14:47:16.227 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41183.log 2024-09-11T14:47:16.227 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29316.log.gz 2024-09-11T14:47:16.227 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41324.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27521.log 2024-09-11T14:47:16.227 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41324.log.gz 2024-09-11T14:47:16.227 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35120.log 2024-09-11T14:47:16.227 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41183.log.gz 2024-09-11T14:47:16.227 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph.tmp-client.admin.27521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31712.log 2024-09-11T14:47:16.227 INFO:teuthology.orchestra.run.smithi181.stderr: 66.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27521.log.gz 2024-09-11T14:47:16.227 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41843.log 2024-09-11T14:47:16.227 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35120.log.gz 2024-09-11T14:47:16.227 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.31712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34406.log 2024-09-11T14:47:16.227 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31712.log.gz 2024-09-11T14:47:16.228 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37926.log 2024-09-11T14:47:16.228 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41843.log.gz 2024-09-11T14:47:16.228 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36656.log 2024-09-11T14:47:16.228 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34406.log.gz 2024-09-11T14:47:16.228 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37926.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-11T14:47:16.228 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37926.log.gz 2024-09-11T14:47:16.228 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33330.log 2024-09-11T14:47:16.228 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36656.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36656.log.gz 2024-09-11T14:47:16.228 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42134.log 2024-09-11T14:47:16.228 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33330.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33330.log.gz 2024-09-11T14:47:16.228 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36874.log 2024-09-11T14:47:16.228 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40679.log 2024-09-11T14:47:16.228 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42134.log: 88.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-11T14:47:16.229 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42134.log.gz 2024-09-11T14:47:16.229 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35181.log 2024-09-11T14:47:16.229 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36874.log.gz 2024-09-11T14:47:16.229 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31783.log 2024-09-11T14:47:16.229 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40679.log.gz 2024-09-11T14:47:16.229 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37399.log 2024-09-11T14:47:16.229 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35181.log.gz 2024-09-11T14:47:16.229 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.31783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36474.log 2024-09-11T14:47:16.229 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31783.log.gz 2024-09-11T14:47:16.229 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35060.log 2024-09-11T14:47:16.229 INFO:teuthology.orchestra.run.smithi181.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.37399.log.gz 2024-09-11T14:47:16.229 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31692.log 2024-09-11T14:47:16.229 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36474.log.gz 2024-09-11T14:47:16.230 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34922.log 2024-09-11T14:47:16.230 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35060.log.gz 2024-09-11T14:47:16.230 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.31692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41759.log 2024-09-11T14:47:16.230 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31692.log.gz 2024-09-11T14:47:16.230 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30613.log 2024-09-11T14:47:16.230 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34922.log.gz 2024-09-11T14:47:16.230 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31104.log 2024-09-11T14:47:16.230 INFO:teuthology.orchestra.run.smithi181.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41759.log.gz 2024-09-11T14:47:16.230 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.30613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41092.log 2024-09-11T14:47:16.230 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30613.log.gz 2024-09-11T14:47:16.230 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.31104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30564.log 2024-09-11T14:47:16.230 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31104.log.gz 2024-09-11T14:47:16.230 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35585.log 2024-09-11T14:47:16.231 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41092.log.gz 2024-09-11T14:47:16.231 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.30564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37270.log 2024-09-11T14:47:16.231 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30564.log.gz 2024-09-11T14:47:16.231 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38752.log 2024-09-11T14:47:16.231 INFO:teuthology.orchestra.run.smithi181.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.35585.log.gz 2024-09-11T14:47:16.231 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30091.log 2024-09-11T14:47:16.231 INFO:teuthology.orchestra.run.smithi181.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.37270.log.gz 2024-09-11T14:47:16.231 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35100.log 49.6% 2024-09-11T14:47:16.231 INFO:teuthology.orchestra.run.smithi181.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38752.log.gz 2024-09-11T14:47:16.231 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.30091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32063.log 2024-09-11T14:47:16.231 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30091.log.gz 2024-09-11T14:47:16.231 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35100.log: gzip -5 --verbose 50.0% -- -- replaced with /var/log/ceph/ceph-client.admin.35100.log.gz /var/log/ceph/ceph-client.admin.38395.log 2024-09-11T14:47:16.231 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-11T14:47:16.231 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35229.log 2024-09-11T14:47:16.232 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32063.log.gz 2024-09-11T14:47:16.232 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28741.log 2024-09-11T14:47:16.232 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38395.log.gz 2024-09-11T14:47:16.232 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40635.log 2024-09-11T14:47:16.232 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35229.log.gz 2024-09-11T14:47:16.232 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.28741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40554.log 2024-09-11T14:47:16.232 INFO:teuthology.orchestra.run.smithi181.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.28741.log.gz 2024-09-11T14:47:16.232 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29590.log 2024-09-11T14:47:16.232 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40635.log.gz 2024-09-11T14:47:16.232 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38485.log 2024-09-11T14:47:16.232 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40554.log.gz 2024-09-11T14:47:16.232 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37528.log 2024-09-11T14:47:16.232 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29590.log.gz 2024-09-11T14:47:16.233 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29128.log 2024-09-11T14:47:16.233 INFO:teuthology.orchestra.run.smithi181.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.38485.log.gz 2024-09-11T14:47:16.233 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30515.log 2024-09-11T14:47:16.233 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37528.log.gz 2024-09-11T14:47:16.233 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37359.log 2024-09-11T14:47:16.233 INFO:teuthology.orchestra.run.smithi181.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29128.log.gz 2024-09-11T14:47:16.233 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.30515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34882.log 2024-09-11T14:47:16.233 INFO:teuthology.orchestra.run.smithi181.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30515.log.gz 2024-09-11T14:47:16.233 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33997.log 2024-09-11T14:47:16.233 INFO:teuthology.orchestra.run.smithi181.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.37359.log.gz 2024-09-11T14:47:16.233 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29356.log 2024-09-11T14:47:16.233 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34882.log.gz 2024-09-11T14:47:16.233 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33519.log 2024-09-11T14:47:16.234 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33997.log.gz 2024-09-11T14:47:16.234 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39221.log 2024-09-11T14:47:16.234 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29356.log.gz 2024-09-11T14:47:16.234 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41717.log 2024-09-11T14:47:16.234 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33519.log.gz 2024-09-11T14:47:16.234 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33917.log 2024-09-11T14:47:16.234 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39221.log.gz 2024-09-11T14:47:16.234 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39359.log 2024-09-11T14:47:16.234 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41717.log.gz 2024-09-11T14:47:16.234 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36705.log 2024-09-11T14:47:16.234 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33917.log.gz 2024-09-11T14:47:16.234 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33728.log 2024-09-11T14:47:16.234 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39359.log.gz 2024-09-11T14:47:16.234 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36705.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36705.log.gz 2024-09-11T14:47:16.234 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38206.log 2024-09-11T14:47:16.235 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37439.log 2024-09-11T14:47:16.235 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33728.log.gz 2024-09-11T14:47:16.235 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35447.log 2024-09-11T14:47:16.235 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38206.log.gz 2024-09-11T14:47:16.235 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37439.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37439.log.gz 2024-09-11T14:47:16.235 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41633.log 2024-09-11T14:47:16.235 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36236.log 2024-09-11T14:47:16.235 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35447.log.gz 2024-09-11T14:47:16.235 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41549.log 50.0% 2024-09-11T14:47:16.235 INFO:teuthology.orchestra.run.smithi181.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41633.log.gz 2024-09-11T14:47:16.235 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33001.log 2024-09-11T14:47:16.235 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36236.log.gz 2024-09-11T14:47:16.235 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37180.log 2024-09-11T14:47:16.235 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41549.log.gz 2024-09-11T14:47:16.236 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32629.log 2024-09-11T14:47:16.236 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33001.log.gz 2024-09-11T14:47:16.236 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29778.log 2024-09-11T14:47:16.236 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37180.log.gz 2024-09-11T14:47:16.236 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32238.log 2024-09-11T14:47:16.236 INFO:teuthology.orchestra.run.smithi181.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32629.log.gz 2024-09-11T14:47:16.236 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43229.log 2024-09-11T14:47:16.236 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29778.log.gz 2024-09-11T14:47:16.236 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29876.log 2024-09-11T14:47:16.236 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32238.log.gz 2024-09-11T14:47:16.236 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43229.log: gzip -5 --verbose 49.6% -- -- replaced with /var/log/ceph/ceph-client.admin.43229.log.gz 2024-09-11T14:47:16.236 INFO:teuthology.orchestra.run.smithi181.stderr: /var/log/ceph/ceph-client.admin.32769.log 2024-09-11T14:47:16.236 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42935.log 2024-09-11T14:47:16.237 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29876.log.gz 2024-09-11T14:47:16.237 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40056.log 2024-09-11T14:47:16.237 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32769.log.gz 2024-09-11T14:47:16.237 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34455.log 2024-09-11T14:47:16.237 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42935.log.gz 2024-09-11T14:47:16.237 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40056.log: gzip 50.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.40056.log.gz --verbose 2024-09-11T14:47:16.237 INFO:teuthology.orchestra.run.smithi181.stderr: -- /var/log/ceph/ceph-client.admin.28567.log 2024-09-11T14:47:16.237 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29019.log 2024-09-11T14:47:16.237 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34455.log.gz 2024-09-11T14:47:16.237 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.28567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33599.log 2024-09-11T14:47:16.237 INFO:teuthology.orchestra.run.smithi181.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28567.log.gz 2024-09-11T14:47:16.237 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37717.log 2024-09-11T14:47:16.237 INFO:teuthology.orchestra.run.smithi181.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29019.log.gz 2024-09-11T14:47:16.237 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34971.log 2024-09-11T14:47:16.238 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33599.log.gz 2024-09-11T14:47:16.238 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41985.log 2024-09-11T14:47:16.238 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37717.log.gz 2024-09-11T14:47:16.238 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29020.log 2024-09-11T14:47:16.238 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34971.log.gz 2024-09-11T14:47:16.238 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41985.log: gzip -5 --verbose -- 50.0% /var/log/ceph/ceph-client.admin.39660.log -- replaced with /var/log/ceph/ceph-client.admin.41985.log.gz 2024-09-11T14:47:16.238 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-11T14:47:16.238 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40416.log 2024-09-11T14:47:16.238 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29020.log.gz 2024-09-11T14:47:16.238 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34126.log 2024-09-11T14:47:16.238 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39660.log.gz 2024-09-11T14:47:16.238 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37886.log 2024-09-11T14:47:16.238 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40416.log.gz 2024-09-11T14:47:16.238 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42893.log 2024-09-11T14:47:16.238 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34126.log.gz 2024-09-11T14:47:16.239 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39052.log 2024-09-11T14:47:16.239 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37886.log.gz 2024-09-11T14:47:16.239 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29877.log 2024-09-11T14:47:16.239 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42893.log.gz 2024-09-11T14:47:16.239 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41141.log 2024-09-11T14:47:16.239 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39052.log.gz 2024-09-11T14:47:16.239 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38659.log 2024-09-11T14:47:16.239 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29877.log.gz 2024-09-11T14:47:16.239 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36005.log 2024-09-11T14:47:16.239 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41141.log.gz 2024-09-11T14:47:16.239 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35020.log 2024-09-11T14:47:16.239 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38659.log.gz 2024-09-11T14:47:16.239 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36005.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36005.log.gz 2024-09-11T14:47:16.240 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32531.log 2024-09-11T14:47:16.240 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35020.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35020.log.gz 2024-09-11T14:47:16.240 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33421.log 2024-09-11T14:47:16.240 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29875.log 2024-09-11T14:47:16.240 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32531.log.gz 2024-09-11T14:47:16.240 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32411.log 2024-09-11T14:47:16.240 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33421.log.gz 2024-09-11T14:47:16.240 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40594.log 2024-09-11T14:47:16.240 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29875.log.gz 2024-09-11T14:47:16.240 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42133.log 2024-09-11T14:47:16.240 INFO:teuthology.orchestra.run.smithi181.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.32411.log.gz 2024-09-11T14:47:16.240 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40594.log: gzip 50.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.40594.log.gz --verbose 2024-09-11T14:47:16.240 INFO:teuthology.orchestra.run.smithi181.stderr: -- /var/log/ceph/ceph-client.admin.42263.log 2024-09-11T14:47:16.240 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42133.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-11T14:47:16.241 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42133.log.gz 2024-09-11T14:47:16.241 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39569.log 2024-09-11T14:47:16.241 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42263.log.gz 2024-09-11T14:47:16.241 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34664.log 2024-09-11T14:47:16.241 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39569.log: gzip -5 --verbose 49.6% -- /var/log/ceph/ceph-client.admin.42767.log -- replaced with /var/log/ceph/ceph-client.admin.39569.log.gz 2024-09-11T14:47:16.241 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-11T14:47:16.241 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32580.log 2024-09-11T14:47:16.241 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34664.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34664.log.gz 2024-09-11T14:47:16.241 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42767.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42767.log.gz 2024-09-11T14:47:16.241 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31454.log 2024-09-11T14:47:16.241 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36425.log 2024-09-11T14:47:16.241 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32580.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32580.log.gz 2024-09-11T14:47:16.241 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.31454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33141.log 2024-09-11T14:47:16.241 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31454.log.gz 2024-09-11T14:47:16.242 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34842.log 2024-09-11T14:47:16.242 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36425.log.gz 2024-09-11T14:47:16.242 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41801.log 2024-09-11T14:47:16.242 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33141.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33141.log.gz 2024-09-11T14:47:16.242 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29186.log 2024-09-11T14:47:16.242 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34842.log.gz 2024-09-11T14:47:16.242 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42431.log 2024-09-11T14:47:16.242 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41801.log.gz 2024-09-11T14:47:16.242 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43187.log 2024-09-11T14:47:16.242 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29186.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29186.log.gz 2024-09-11T14:47:16.242 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39918.log 2024-09-11T14:47:16.242 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42431.log.gz 2024-09-11T14:47:16.242 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33050.log 2024-09-11T14:47:16.242 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43187.log.gz 2024-09-11T14:47:16.242 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36754.log 2024-09-11T14:47:16.243 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39918.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39918.log.gz 2024-09-11T14:47:16.243 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30838.log 2024-09-11T14:47:16.243 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33050.log.gz 2024-09-11T14:47:16.243 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37668.log 2024-09-11T14:47:16.243 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36754.log.gz 2024-09-11T14:47:16.243 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33281.log 2024-09-11T14:47:16.243 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.30838.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.30838.log.gz 2024-09-11T14:47:16.243 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37668.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37668.log.gz 2024-09-11T14:47:16.243 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29018.log 2024-09-11T14:47:16.243 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36096.log 2024-09-11T14:47:16.243 INFO:teuthology.orchestra.run.smithi181.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33281.log.gz 2024-09-11T14:47:16.243 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33819.log 2024-09-11T14:47:16.243 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29018.log.gz 2024-09-11T14:47:16.243 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38529.log 2024-09-11T14:47:16.243 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36096.log.gz 2024-09-11T14:47:16.244 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42511.log 2024-09-11T14:47:16.244 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33819.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33819.log.gz 2024-09-11T14:47:16.244 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35676.log 2024-09-11T14:47:16.244 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38529.log.gz 2024-09-11T14:47:16.244 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34584.log 2024-09-11T14:47:16.244 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42511.log.gz 2024-09-11T14:47:16.244 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28398.log 2024-09-11T14:47:16.244 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35676.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35676.log.gz 2024-09-11T14:47:16.244 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37012.log 2024-09-11T14:47:16.244 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34584.log.gz 2024-09-11T14:47:16.244 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.28398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28834.log 2024-09-11T14:47:16.244 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28398.log.gz 2024-09-11T14:47:16.244 INFO:teuthology.orchestra.run.smithi181.stderr:gzip/var/log/ceph/ceph-client.admin.37012.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34360.log 2024-09-11T14:47:16.244 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37012.log.gz 2024-09-11T14:47:16.244 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.28834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34500.log 2024-09-11T14:47:16.245 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28834.log.gz 2024-09-11T14:47:16.245 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43019.log 2024-09-11T14:47:16.245 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34360.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34360.log.gz 2024-09-11T14:47:16.245 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34500.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34500.log.gz 2024-09-11T14:47:16.245 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35907.log 2024-09-11T14:47:16.245 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32860.log 2024-09-11T14:47:16.245 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43019.log.gz 2024-09-11T14:47:16.245 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42725.log 2024-09-11T14:47:16.245 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35907.log.gz 2024-09-11T14:47:16.245 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36285.log 2024-09-11T14:47:16.245 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32860.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32860.log.gz 2024-09-11T14:47:16.245 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42725.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42725.log.gz 2024-09-11T14:47:16.245 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43061.log 2024-09-11T14:47:16.246 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36285.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36285.log.gz 2024-09-11T14:47:16.246 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32720.log 2024-09-11T14:47:16.246 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32949.log 2024-09-11T14:47:16.246 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43061.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43061.log.gz 2024-09-11T14:47:16.246 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29017.log 2024-09-11T14:47:16.246 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32720.log.gz 2024-09-11T14:47:16.246 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32949.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-11T14:47:16.246 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32949.log.gz 2024-09-11T14:47:16.246 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35816.log 2024-09-11T14:47:16.246 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29017.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29017.log.gz 2024-09-11T14:47:16.246 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35298.log 2024-09-11T14:47:16.246 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35816.log: gzip -5 --verbose -- 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35816.log.gz 2024-09-11T14:47:16.246 INFO:teuthology.orchestra.run.smithi181.stderr: /var/log/ceph/ceph-client.admin.42155.log 2024-09-11T14:47:16.246 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35298.log: 86.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-11T14:47:16.247 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35298.log.gz 2024-09-11T14:47:16.247 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39618.log 2024-09-11T14:47:16.247 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35865.log 2024-09-11T14:47:16.247 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42155.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42155.log.gz 2024-09-11T14:47:16.247 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36145.log 2024-09-11T14:47:16.247 INFO:teuthology.orchestra.run.smithi181.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.39618.log.gz 2024-09-11T14:47:16.247 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38874.log 2024-09-11T14:47:16.247 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35865.log.gz 2024-09-11T14:47:16.247 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42135.log 2024-09-11T14:47:16.247 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36145.log.gz 2024-09-11T14:47:16.247 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28876.log 2024-09-11T14:47:16.247 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38874.log.gz 2024-09-11T14:47:16.247 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32491.log 2024-09-11T14:47:16.248 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42135.log.gz 2024-09-11T14:47:16.248 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.28876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38834.log 2024-09-11T14:47:16.248 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28876.log.gz 2024-09-11T14:47:16.248 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37092.log 2024-09-11T14:47:16.248 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32491.log.gz 2024-09-11T14:47:16.248 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41282.log 2024-09-11T14:47:16.248 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38834.log.gz 2024-09-11T14:47:16.248 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43311.log 2024-09-11T14:47:16.248 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37092.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37092.log.gz 2024-09-11T14:47:16.248 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34175.log 2024-09-11T14:47:16.248 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41282.log.gz 2024-09-11T14:47:16.248 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39709.log 2024-09-11T14:47:16.249 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43311.log.gz 2024-09-11T14:47:16.249 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40861.log 2024-09-11T14:47:16.249 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34175.log.gz 2024-09-11T14:47:16.249 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36963.log 2024-09-11T14:47:16.249 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39709.log.gz 2024-09-11T14:47:16.249 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34315.log 2024-09-11T14:47:16.249 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40861.log.gz 2024-09-11T14:47:16.249 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33237.log 2024-09-11T14:47:16.249 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36963.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36963.log.gz 2024-09-11T14:47:16.249 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39439.log 2024-09-11T14:47:16.249 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34315.log.gz 2024-09-11T14:47:16.249 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33237.log: gzip -5 --verbose 50.0% -- -- replaced with /var/log/ceph/ceph-client.admin.33237.log.gz /var/log/ceph/ceph-client.admin.40376.log 2024-09-11T14:47:16.249 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-11T14:47:16.249 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40505.log 2024-09-11T14:47:16.250 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39439.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39439.log.gz 2024-09-11T14:47:16.250 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40763.log 2024-09-11T14:47:16.250 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40376.log.gz 2024-09-11T14:47:16.250 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34802.log 2024-09-11T14:47:16.250 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40505.log.gz 2024-09-11T14:47:16.250 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39093.log 2024-09-11T14:47:16.250 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40763.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40763.log.gz 2024-09-11T14:47:16.250 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39181.log 2024-09-11T14:47:16.250 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34802.log.gz 2024-09-11T14:47:16.250 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37123.log 2024-09-11T14:47:16.250 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39093.log.gz 2024-09-11T14:47:16.250 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41984.log 2024-09-11T14:47:16.250 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39181.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39181.log.gz 2024-09-11T14:47:16.251 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37757.log 2024-09-11T14:47:16.251 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37123.log.gz 2024-09-11T14:47:16.251 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41885.log 2024-09-11T14:47:16.251 INFO:teuthology.orchestra.run.smithi181.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41984.log.gz 2024-09-11T14:47:16.251 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29632.log 2024-09-11T14:47:16.251 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37757.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37757.log.gz 2024-09-11T14:47:16.251 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42451.log 2024-09-11T14:47:16.251 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41885.log.gz 2024-09-11T14:47:16.251 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42599.log 2024-09-11T14:47:16.251 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29632.log.gz 2024-09-11T14:47:16.251 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30962.log 2024-09-11T14:47:16.251 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42451.log.gz 2024-09-11T14:47:16.251 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42599.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42599.log.gz 2024-09-11T14:47:16.251 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37577.log 2024-09-11T14:47:16.252 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35487.log 2024-09-11T14:47:16.252 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.30962.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30962.log.gz 2024-09-11T14:47:16.252 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29506.log 2024-09-11T14:47:16.252 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37577.log.gz 2024-09-11T14:47:16.252 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42977.log 2024-09-11T14:47:16.252 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35487.log.gz 2024-09-11T14:47:16.252 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36834.log 2024-09-11T14:47:16.252 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29506.log.gz 2024-09-11T14:47:16.252 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36047.log 2024-09-11T14:47:16.252 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42977.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42977.log.gz 2024-09-11T14:47:16.252 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42851.log 2024-09-11T14:47:16.252 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36834.log.gz 2024-09-11T14:47:16.252 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33093.log 2024-09-11T14:47:16.253 INFO:teuthology.orchestra.run.smithi181.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.36047.log.gz 2024-09-11T14:47:16.253 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42851.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42851.log.gz 2024-09-11T14:47:16.253 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41987.log 2024-09-11T14:47:16.253 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32196.log 2024-09-11T14:47:16.253 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33093.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33093.log.gz 2024-09-11T14:47:16.253 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39479.log 49.6% 2024-09-11T14:47:16.253 INFO:teuthology.orchestra.run.smithi181.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41987.log.gz 2024-09-11T14:47:16.253 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32258.log 2024-09-11T14:47:16.253 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32196.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32196.log.gz 2024-09-11T14:47:16.253 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39479.log: gzip -5 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39479.log.gz 2024-09-11T14:47:16.253 INFO:teuthology.orchestra.run.smithi181.stderr: --verbose -- /var/log/ceph/ceph-client.admin.30133.log 2024-09-11T14:47:16.253 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39139.log 2024-09-11T14:47:16.253 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32258.log.gz 2024-09-11T14:47:16.253 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.30133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43103.log 2024-09-11T14:47:16.254 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30133.log.gz 2024-09-11T14:47:16.254 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40436.log 2024-09-11T14:47:16.254 INFO:teuthology.orchestra.run.smithi181.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.39139.log.gz 2024-09-11T14:47:16.254 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43103.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43103.log.gz 2024-09-11T14:47:16.254 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38794.log 2024-09-11T14:47:16.254 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30922.log 2024-09-11T14:47:16.254 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40436.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40436.log.gz 2024-09-11T14:47:16.254 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38794.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38794.log.gz 2024-09-11T14:47:16.254 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37975.log 2024-09-11T14:47:16.254 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39758.log 2024-09-11T14:47:16.254 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.30922.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30922.log.gz 2024-09-11T14:47:16.254 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35536.log 2024-09-11T14:47:16.254 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37975.log.gz 2024-09-11T14:47:16.255 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39399.log 2024-09-11T14:47:16.255 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39758.log.gz 2024-09-11T14:47:16.255 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35536.log: gzip -5 --verbose 50.0% -- -- replaced with /var/log/ceph/ceph-client.admin.35536.log.gz /var/log/ceph/ceph-client.admin.28595.log 2024-09-11T14:47:16.255 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-11T14:47:16.255 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33639.log 2024-09-11T14:47:16.255 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39399.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39399.log.gz 2024-09-11T14:47:16.255 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.28595.log: gzip -5 --verbose 50.0% -- -- replaced with /var/log/ceph/ceph-client.admin.28595.log.gz /var/log/ceph/ceph-client.admin.42303.log 2024-09-11T14:47:16.255 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-11T14:47:16.255 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35956.log 2024-09-11T14:47:16.255 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33639.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33639.log.gz 2024-09-11T14:47:16.255 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34266.log 2024-09-11T14:47:16.255 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42303.log.gz 2024-09-11T14:47:16.255 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34078.log 2024-09-11T14:47:16.255 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35956.log.gz 2024-09-11T14:47:16.256 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31008.log 2024-09-11T14:47:16.256 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34266.log.gz 2024-09-11T14:47:16.256 INFO:teuthology.orchestra.run.smithi181.stderr:gzip/var/log/ceph/ceph-client.admin.34078.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.33379.log 2024-09-11T14:47:16.256 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34078.log.gz 2024-09-11T14:47:16.256 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.31008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33190.log 2024-09-11T14:47:16.256 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31008.log.gz 2024-09-11T14:47:16.256 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40336.log 2024-09-11T14:47:16.256 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33379.log.gz 2024-09-11T14:47:16.256 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5/var/log/ceph/ceph-client.admin.33190.log: --verbose -- /var/log/ceph/ceph-client.admin.34713.log 2024-09-11T14:47:16.256 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33190.log.gz 2024-09-11T14:47:16.256 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40336.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40336.log.gz 2024-09-11T14:47:16.256 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37619.log 2024-09-11T14:47:16.256 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30395.log 2024-09-11T14:47:16.257 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34713.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34713.log.gz 2024-09-11T14:47:16.257 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28792.log 2024-09-11T14:47:16.257 INFO:teuthology.orchestra.run.smithi181.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.37619.log.gz 2024-09-11T14:47:16.257 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.30395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38710.log 2024-09-11T14:47:16.257 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30395.log.gz 2024-09-11T14:47:16.257 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.28792.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28792.log.gz 2024-09-11T14:47:16.257 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35767.log 2024-09-11T14:47:16.257 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41043.log 2024-09-11T14:47:16.257 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38710.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38710.log.gz 2024-09-11T14:47:16.257 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.35767.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35767.log.gz 2024-09-11T14:47:16.257 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42472.log 2024-09-11T14:47:16.257 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41464.log 2024-09-11T14:47:16.257 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41043.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41043.log.gz 2024-09-11T14:47:16.257 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42472.log: gzip -5 --verbose 49.6% -- /var/log/ceph/ceph-client.admin.36914.log -- replaced with /var/log/ceph/ceph-client.admin.42472.log.gz 2024-09-11T14:47:16.258 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-11T14:47:16.258 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30355.log 2024-09-11T14:47:16.258 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41464.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41464.log.gz 2024-09-11T14:47:16.258 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29187.log 2024-09-11T14:47:16.258 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36914.log.gz 2024-09-11T14:47:16.258 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.30355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29464.log 2024-09-11T14:47:16.258 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30355.log.gz 2024-09-11T14:47:16.258 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31153.log 2024-09-11T14:47:16.258 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29187.log.gz 2024-09-11T14:47:16.258 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.29464.log: /var/log/ceph/ceph-client.admin.30705.log 2024-09-11T14:47:16.258 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29464.log.gz 2024-09-11T14:47:16.258 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.31153.log: gzip -5 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31153.log.gz 2024-09-11T14:47:16.258 INFO:teuthology.orchestra.run.smithi181.stderr: --verbose -- /var/log/ceph/ceph-client.admin.33679.log 2024-09-11T14:47:16.259 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.30705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32811.log 2024-09-11T14:47:16.259 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30705.log.gz 2024-09-11T14:47:16.259 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39967.log 2024-09-11T14:47:16.259 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33679.log.gz 2024-09-11T14:47:16.259 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37319.log 2024-09-11T14:47:16.259 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32811.log.gz 2024-09-11T14:47:16.259 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36614.log 2024-09-11T14:47:16.259 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39967.log.gz 2024-09-11T14:47:16.259 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34762.log 2024-09-11T14:47:16.259 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37319.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37319.log.gz 2024-09-11T14:47:16.259 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37479.log 2024-09-11T14:47:16.259 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36614.log.gz 2024-09-11T14:47:16.259 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31552.log 2024-09-11T14:47:16.259 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34762.log.gz 2024-09-11T14:47:16.260 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37777.log 2024-09-11T14:47:16.260 INFO:teuthology.orchestra.run.smithi181.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.37479.log.gz 2024-09-11T14:47:16.260 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.31552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32104.log 2024-09-11T14:47:16.260 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31552.log.gz 2024-09-11T14:47:16.260 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29357.log 2024-09-11T14:47:16.260 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37777.log.gz 2024-09-11T14:47:16.260 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31294.log 2024-09-11T14:47:16.260 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32104.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32104.log.gz 2024-09-11T14:47:16.260 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29357.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29357.log.gz 2024-09-11T14:47:16.260 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39261.log 2024-09-11T14:47:16.260 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37842.log 2024-09-11T14:47:16.260 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.31294.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31294.log.gz 2024-09-11T14:47:16.260 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39261.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39261.log.gz 2024-09-11T14:47:16.261 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37842.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37842.log.gz 2024-09-11T14:47:16.347 INFO:teuthology.orchestra.run.smithi181.stderr: 89.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-11T14:47:26.966 INFO:teuthology.orchestra.run.smithi181.stderr: 91.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-11T14:47:27.170 INFO:teuthology.orchestra.run.smithi203.stderr: 91.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-11T14:47:27.794 INFO:teuthology.orchestra.run.smithi181.stderr: 91.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-11T14:47:27.796 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-11T14:47:27.796 INFO:teuthology.orchestra.run.smithi181.stderr:real 0m11.711s 2024-09-11T14:47:27.796 INFO:teuthology.orchestra.run.smithi181.stderr:user 0m22.067s 2024-09-11T14:47:27.796 INFO:teuthology.orchestra.run.smithi181.stderr:sys 0m0.904s 2024-09-11T14:47:30.504 INFO:teuthology.orchestra.run.smithi203.stderr: 91.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-11T14:47:30.505 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-11T14:47:30.505 INFO:teuthology.orchestra.run.smithi203.stderr:real 0m14.375s 2024-09-11T14:47:30.505 INFO:teuthology.orchestra.run.smithi203.stderr:user 0m24.650s 2024-09-11T14:47:30.505 INFO:teuthology.orchestra.run.smithi203.stderr:sys 0m0.764s 2024-09-11T14:47:30.505 INFO:tasks.ceph:Archiving logs... 2024-09-11T14:47:30.505 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/log/ceph to /home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901371/remote/smithi181/log 2024-09-11T14:47:30.506 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T14:47:34.716 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/log/ceph to /home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901371/remote/smithi203/log 2024-09-11T14:47:34.717 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T14:47:39.172 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-11T14:47:39.181 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-11T14:47:39.182 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T14:47:39.185 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T14:47:39.267 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-11T14:47:39.276 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-11T14:47:39.277 DEBUG:teuthology.orchestra.run.smithi181:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T14:47:39.279 DEBUG:teuthology.orchestra.run.smithi203:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T14:47:39.293 INFO:teuthology.orchestra.run.smithi181.stderr:bash: line 1: ntpq: command not found 2024-09-11T14:47:39.296 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-09-11T14:47:39.298 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T14:47:39.298 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-09-11T14:47:39.298 INFO:teuthology.orchestra.run.smithi181.stdout:^* hv01.front.sepia.ceph.com 3 6 377 64 +869us[ +830us] +/- 54ms 2024-09-11T14:47:39.298 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 64 -597us[ -636us] +/- 53ms 2024-09-11T14:47:39.298 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 64 -518us[ -558us] +/- 84ms 2024-09-11T14:47:39.298 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T14:47:39.300 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T14:47:39.300 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-09-11T14:47:39.300 INFO:teuthology.orchestra.run.smithi203.stdout:^* hv01.front.sepia.ceph.com 3 6 377 1 +882us[ +841us] +/- 55ms 2024-09-11T14:47:39.300 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 64 -590us[ -628us] +/- 53ms 2024-09-11T14:47:39.300 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 0 -529us[ -529us] +/- 85ms 2024-09-11T14:47:39.301 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T14:47:39.301 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-11T14:47:39.311 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-11T14:47:39.311 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-11T14:47:39.320 DEBUG:teuthology.orchestra.run.smithi181:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-11T14:47:39.411 DEBUG:teuthology.orchestra.run.smithi203:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-11T14:47:39.478 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-11T14:47:39.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T14:47:39.507 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-11T14:47:39.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T14:47:39.536 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-11T14:47:39.545 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-11T14:47:39.555 INFO:teuthology.task.internal:Duration was 808.905934 seconds 2024-09-11T14:47:39.556 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-11T14:47:39.564 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-11T14:47:39.564 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T14:47:39.567 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T14:47:39.604 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T14:47:39.616 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T14:47:39.947 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-11T14:47:39.947 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi181.front.sepia.ceph.com 2024-09-11T14:47:39.947 DEBUG:teuthology.orchestra.run.smithi181:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T14:47:39.968 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi203.front.sepia.ceph.com 2024-09-11T14:47:39.968 DEBUG:teuthology.orchestra.run.smithi203:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T14:47:40.013 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-11T14:47:40.013 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T14:47:40.016 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T14:47:40.081 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-11T14:47:40.081 DEBUG:teuthology.orchestra.run.smithi181:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T14:47:40.084 DEBUG:teuthology.orchestra.run.smithi203:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T14:47:40.217 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-11T14:47:40.228 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-11T14:47:40.228 DEBUG:teuthology.orchestra.run.smithi181:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T14:47:40.253 DEBUG:teuthology.orchestra.run.smithi203:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T14:47:40.279 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-11T14:47:40.288 DEBUG:teuthology.orchestra.run.smithi181:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T14:47:40.296 DEBUG:teuthology.orchestra.run.smithi203:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T14:47:40.317 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = core 2024-09-11T14:47:40.342 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = core 2024-09-11T14:47:40.361 DEBUG:teuthology.orchestra.run.smithi181:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T14:47:40.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T14:47:40.395 DEBUG:teuthology.orchestra.run.smithi203:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T14:47:40.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T14:47:40.415 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-11T14:47:40.427 INFO:teuthology.task.internal:Transferring archived files... 2024-09-11T14:47:40.428 DEBUG:teuthology.misc:Transferring archived files from smithi181:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901371/remote/smithi181 2024-09-11T14:47:40.428 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T14:47:40.469 DEBUG:teuthology.misc:Transferring archived files from smithi203:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901371/remote/smithi203 2024-09-11T14:47:40.469 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T14:47:40.503 INFO:teuthology.task.internal:Removing archive directory... 2024-09-11T14:47:40.503 DEBUG:teuthology.orchestra.run.smithi181:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T14:47:40.509 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T14:47:40.552 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-11T14:47:40.562 INFO:teuthology.task.internal:Not uploading archives. 2024-09-11T14:47:40.562 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-11T14:47:40.571 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-11T14:47:40.571 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T14:47:40.574 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T14:47:40.586 INFO:teuthology.orchestra.run.smithi181.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 14:47 /home/ubuntu/cephtest 2024-09-11T14:47:40.607 INFO:teuthology.orchestra.run.smithi203.stdout: 265730 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 14:47 /home/ubuntu/cephtest 2024-09-11T14:47:40.607 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-11T14:47:40.617 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-11T14:47:40.640 INFO:teuthology.run:Summary data: description: crimson-rados/thrash_simple/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore thrashers/simple thrashosds-health workloads/admin_socket_objecter_requests} duration: 808.9059340953827 flavor: crimson owner: scheduled_matan@teuthology success: true 2024-09-11T14:47:40.640 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T14:47:40.720 INFO:teuthology.run:pass